1 Reply Latest reply on Oct 28, 2021 8:37 PM by Shabari Nath Kadavendi

    How to achieve Batch/Incremental pull via API ( Multiple records in one Response like millions of records)

    Shabari Nath Kadavendi Active Member

      Hi Guys,


      Can anyone please help me with the below problem at the earliest?


      I have a requirement to pull data from a variety of API's. The response of the API is not one-one. Each request will have 1000's of records and I need to load the response of those 1000 records in database table.



      1)  How could I achieve this via CAI( Service Connector) or CDI (Rest V2 Connector + Webservices/Business Services)?

      2)  I am unable to extract multiple records from the response while loading into database. It inserts only the first available entity in the response (XML or JSON) after the service connector.

      3) When I test the service connector for results, it was success full and shows multiple records. but unable to parse the response from Service Connector in Process.

      4) i've tried to use Output as Entire Response  or Entire Response as Attachment or Individual property.

      5) Nothing worked out. How could I parse the response from Service Connector.


      Does CAI handles batch pulls? like for each request, can it extract and parse 1000+ records? if so how to achieve it? if not what is the solution? CDI? if CDI, please elaborate how this can be achieved in CDI?



      Also, please help me provide a solution on how to achieve this batch pull for Initial Full Load and Incremental Delta pulls from both the API in CAI as well CDI?


      When I use Entire Response as Attachment in the service connector and when I run the process, where does the payload (JSON or XML gets saved?


      if it stores anywhere on the secure agent, i would like to know so I can use that to parse and load in the daytabase.



      Also, my source API has millions of records and I need to pull everything for the first load and the API has a response limit of only 250 per request. so how can I achieve this in Informatica either CAI or CDI (like how can I achieve the pagination or looping process) or how can I achieve parallelism like calling 1000's of API calls. is having a 1000's of parallel API calls better or get a one full batch request in one payload or break the huge payload into multiple chunks and process those chunks in parallel. if so, please elaborate your solution.





      Please help me by elaborated instructions if possible.