1 Reply Latest reply on Aug 29, 2018 5:00 PM by Frank Cunningham

    Kafka partitioned topics with IDR

    Marija Vas New Member

      We are using IDR 9.7.0 HF2 to output CDC traffic into kafka topics.

       

      We are using USE_TABLE_NAME to generate one topic per table.

      We would like to be able to have one topic per source table(which we already do have), but data should be sent to different topic partitions meaningfully... eg by hash of Primary Key columns from relevant source table.

       

      For now anything we tried only resulted in:
      - having random round-robin meaninglessly partitioned data in kafka topics
      OR
      - having one partition per topic
      None of the above are our desired outcome.

      Is there a setting in IDR which would enable us to have topics partitioned by hash of table primary keys in the config?

        • 1. Re: Kafka partitioned topics with IDR
          Frank Cunningham Active Member

          Hi Marija,

           

          At this time there is no option to allow for messages to be sent to a partition based on a column value.

           

          There are additional Applier parameters for a Kafka target that will allow to send to a partition ID for that topic but I believe you are asking for a more granular partitioning scheme, if I am correct.

           

          If you would like to see an addition to the product, please open a case with Informatica Global Customer Support and we can gather more information on a possible Feature Request.

           

          Thank you!