8 Replies Latest reply on May 24, 2020 6:45 AM by Venkat Neppali

    Performance issue with Joiner transformation while loading 2.3 million records

    Venkat Neppali Active Member

      Hi All,

       

      I am loading data from 9 XML files in indirect mode and total number of records are 2.3 million records in one view and other view its only 9 records as its 9 files.

      Process is taking around 1.30 minutes if load is less load on server and if it is more then it is taking around 3 hours.

       

      Enviornment - Powecenter 10.1.1

       

      As per log, I can see that Joiner transformation is taking time.

       

      Sorted input is selected for joiner as source is same and master pipeline have only 9 records and details pipe line has 2.3 million records.

      Joiner condition column is integer column.

      Created one partition point at joiner as well

       

      Would like to improve performance of this process. Please share your suggestions/Inputs for same.

       

      ***** RUN INFO FOR TGT LOAD ORDER GROUP [1], CONCURRENT SET [1] *****

      Thread [READER_1_1_1] created for [the read stage] of partition point [XML_TRANSFORMATION] has completed.

      Total Run Time = [18.333854] secs

      Total Idle Time = [2.984592] secs

      Busy Percentage = [83.720868]

      Thread [TRANSF_1_1_1] created for [the transformation stage] of partition point [XML_TRANSFORMATION] has completed.

      Total Run Time = [4872.297482] secs

      Total Idle Time = [4865.374909] secs

      Busy Percentage = [0.142080]

      Thread work time breakdown:

      JNR_TRANSFORMATION: 99.876442 percent

      EXP_TRANSFORMATION: 0.041186 percent

      JNR_TRANSFORMATION1: 0.082372 percent

      Thread [TRANSF_1_6_1] created for [the transformation stage] of partition point [AGG_TRANSFROMATION] has completed.

      Total Run Time = [2.529216] secs

      Total Idle Time = [2.488863] secs

      Busy Percentage = [1.595477]

      Transformation-specific statistics for this thread were not accurate enough to report.

      Thread [TRANSF_1_8_1] created for [the transformation stage] of partition point [JNR_TRANSFORMATION] has completed.

      Total Run Time = [5654.757800] secs

      Total Idle Time = [0.123175] secs

      Busy Percentage = [99.997822]

      Transformation-specific statistics for this thread were not accurate enough to report.

      Thread [WRITER_1_*_1] created for [the write stage] of partition point [TARGET1, TARGET2] has completed.

      Total Run Time = [5652.429281] secs

      Total Idle Time = [5337.391257] secs

      Busy Percentage = [5.573498]

       

      total number of ports in joiner is 14

       

      Thanks,

      Venkat Neppali.

        This content has been marked as final.  Show 8 replies