6 Replies Latest reply on Aug 3, 2020 8:38 AM by niti rawat

    Real time integration using Informatica Data Quality 10.4

    niti rawat Seasoned Veteran

      Hi All,

       

      Has anybody implemented real time integration/data load in Informatica Data Quality 10.4?

      We are currently working on 10.2 and an upgrade to 10.4 is planned.

      For now the data load is happening in batch mode, but with real time since the latency is reduced to a significant level, we wanted to try and see if that can be achieved in 10.4

       

      Regards,

      Niti

        • 1. Re: Real time integration using Informatica Data Quality 10.4
          user126898 Guru

          What are the source(s) you are trying to use for the real time use case?  Are you looking for a CDC type pull or a streaming?

           

          Thanks,

          Scott

          • 2. Re: Real time integration using Informatica Data Quality 10.4
            niti rawat Seasoned Veteran

            Hi Scott,

             

            We are trying to achieve near time integration.With CDC what i understand is the latency will be high as compared to streaming. Feel free to correct me.

            Also, could you help me understand the usage of Rest API wrt near real time integration if there's any?

             

            Regards,

            Niti

            • 3. Re: Real time integration using Informatica Data Quality 10.4
              user126898 Guru

              CDC and Streaming are technically similar.  CDC centers around pulling data from traditional relational sources (oracle, DB2...) but can achieve near real time pulls.  With a real time mapping pulling from a CDC source as soon as the transaction hits the DB logs we pull the change.  We have customers processing changed records within 4-5 secs of them happening. 

               

              Streaming centers around IoT type data from message ques, sensors, internet feeds..  very quick.

               

              So both are near real time it all depends on what delivery method you prefer.

               

              example.

              oracle CDC dumps changes to kafka and then you have a streaming map reading the kafka topic downstream.  All happening real time.

               

              Inside of streaming products that Informatica offers you can insert Rest calls into the data feed.  The Rest adds some overhead so it needs to be tuned but customers use it all the time.

               

               

              thanks,

              Scott

              • 4. Re: Real time integration using Informatica Data Quality 10.4
                niti rawat Seasoned Veteran

                Our source is a database, called Data Bridge. By CDC source do you mean we need to create a CDC source and a real time mapping.

                I believe CDC is a separate Informatica product.

                How can we construct a real time mapping in Informatica DQ? I have not done such implementations before.

                Can we use just webservice transformation to achieve real time integration in DQ?

                 

                Apologies, if this is getting you confused. We need to implement real time integration on DQ for MDM, and we are using SOAP API's to trigger MDM jobs ( loading data from DQ layer to target applications)

                 

                Regards,

                Niti

                • 5. Re: Real time integration using Informatica Data Quality 10.4
                  user126898 Guru

                  You are correct DQ does not have CDC built in to the product nor does it have any CDC/real time components out of the box.  The use case for imbedding DQ rules into real time feeds, customer export the DQ mapplets and move into a PowerCenter mapping where you do have CDC and real time workflows.

                   

                  The web service may work but will take some configuring. 

                  1) you will need a license for web transformation consumer to call API's

                  2) You will need to create some sort of script that executes the mapping/workflow over and over to simulate the near real time feed.

                   

                  Sadly other products like PwC, Cloud Database Mass Ingestion, Cloud Streaming Mass Ingestion, Data Engineering Stream are designed for this use case.  All of these products can use DQ within the dataflows.

                   

                   

                  thanks,

                  Scott

                  • 6. Re: Real time integration using Informatica Data Quality 10.4
                    niti rawat Seasoned Veteran

                    Hi Scott,

                     

                    Since we already are using webservice transformation to trigger MDM jobs using SOAP UI's, i guess the license condition is already met. As for point#2, i thought if we configure the webservice transformation for REST API, we would not need to schedule a high frequency script so as to capture every change made in the source data. Am i missing something here?

                     

                    Regards,

                    Niti