4 Replies Latest reply on May 4, 2021 2:57 AM by Krishna Gangwar

    IDQ slow profiling

    inuser526279 Guru

      Hi Team,

       

      A table having 125 columns and of 160 rows is taking 20 minutes for profiling. Is it normal?

       

      Can anyone say how much time does it take in your system to profile a table of above volume?

       

      How performance can be enhanced?

       

      Thanks

        • 1. Re: IDQ slow profiling
          Akanksha Rani Seasoned Veteran

          Hi,

           

          Performance can depend on many factors like source ( time taken to read ), target ( time taken to write to PWH ), concurrent jobs ( is the current job getting queued or the total execution time itself is high ), precision of the columns etc. As there are so many factors involved, we do not have any performance benchmarks. One cannot say if the performance is slow or as expected based on the number of rows / columns.

           

          Is the data preview on the same source taking time as well? If you put the same data in a flat file ( say via a mapping from the table to flat file ) & then profile this target flat file, how much time does it take? If it is fast, we can say that the slowness is while reading from the source.

           

          This might need thorough investigation & it is best to open a case for it if indeed you feel that the performance is slow.

           

          Regards,

          Akanksha

          • 2. Re: IDQ slow profiling
            Kathy Erwin New Member

            My experience has been that it is best to profile only a few columns and rules at a time in a single profile in order to get a good response time.

             

            Increasing the number of columns increases the amount of time the profile takes to run. If I choose all columns in a table with many columns, the profile results may never return.

             

            I've found that even profiling all rows in very large tables can be completed in a short amount of time if I limit the number of columns selected. For example, I profiled a table of over 300 million rows using a Snowflake ODBC connection which took only one minute and 47 seconds to run. The key was that I selected only the two columns and one rule that I needed in the profile run.

            • 3. Re: IDQ slow profiling
              Robert Whelan Guru

              A table having 125 columns and of 160 rows is taking 20 minutes for profiling. Is it normal?

               

              Can anyone say how much time does it take in your system to profile a table of above volume?

               

              How performance can be enhanced?

              I'm not sure if 125 columns counts as a wide table, but I'd suggest taking a look at the How-To Guide on profiling wide tables for some general guidance on performance.

              • 4. Re: IDQ slow profiling
                Krishna Gangwar Active Member

                Hi,

                 

                Can you share the profile logs if you are still facing the issue? You can find profile logs under "$INFA_HOME/logs/<node_name>/services/DataIntegrationService/disLogs/profiling" directory on the Informatica server machine.

                 

                Alternatively, you can open a technical case to get further assistance.

                 

                Regards,

                Krishna