7 Replies Latest reply on Mar 24, 2021 6:09 AM by Nithy S

    Error handling in IICS data integration mapping task

    Chetan Sarawade Active Member

      Hello All,

       

      I am a newbie to cloud community. I am working on data integration. I created a Mapping task for loading gz file from the S3 bucket to Redshift. I defined schema at source level. While loading into the target tables a few rows get dropped because of invalid data its happens due layout validation issue. when source data is not matching with schema i defined.
      I am trying to load these rows into the error log file but unable to do that. I enable Advanced session properties
      and select Erro log file as error1, Error log type as fat file, Data column delimiter as comma and enable  Log Source Row Data. but I am getting a blank file.

      Is i am missing something?
      Is there any way that i can load those rows into the error log file??

        • 1. Re: Error handling in IICS data integration mapping task
          Ashwini Ramakrishna Guru

          To access the actual error file, go to the following path and look for the file with task id:

           

           

          <agent_dir>\apps\Data_Integration_Server\data\error

           

          For custom error file name and directory create an MCT and add the Advanced Session Property:

           

          Error Log Type

          Error Log File Directory

          Error Log File Name

          • 2. Re: Error handling in IICS data integration mapping task
            Chetan Sarawade Active Member

            Hi Ashwini
            Thanks for your help.
            I am able to find error log file but the problem is I am getting empty file. There is no record that is inserted into the error log file. Its because
            when the row is rejected at the source, then usually it has to do with the column definition (data type, precision, and scale) defined for one or more columns in the source.

            The column definition may not be able to handle the data in the source and "rejects" the row. Errors in the session log stating as such rows are dropped due to invalid data.

            As we know that rows rejected in the Source do not enter the rejected rows of the session.

            The rows ignored/rejected at the Source Qualifier do not enter the session pipeline.

            I want to capture those rows which got dropped at the level of the source?

            Is there a way I can achieve this?

            • 4. Re: Error handling in IICS data integration mapping task
              Akshaye Shreenithi Kirupa Guru

              Hi Chetan

               

              Please note as of now there is no direct mechanism available to capture source level rejections. Only target level rejections can be captured in error log file. For more information on capturing target level rejections refer this KB ,

              583883 You can have a lookup between source and target to find missing records or use business checks in expression to remove incorrect rows and route them towards error files for further processing.

              • 5. Re: Error handling in IICS data integration mapping task
                Chetan Sarawade Active Member

                Hi, Akshaye thanks for your help.
                how can I capture those records without a direct mechanism? Any idea?

                Can I load those records from the Session log and load it into the file?

                • 6. Re: Error handling in IICS data integration mapping task
                  Akshaye Shreenithi Kirupa Guru

                  Hi Chetan

                   

                  As suggested earlier you can have a lookup between source and target to find missing records and then load it to a flat file to capture these records.

                  • 7. Re: Error handling in IICS data integration mapping task
                    Chetan Sarawade Active Member

                    Hi All,

                    I create one mapping task in IICS Data integration to capture Invalid data into the error log file. In advance Error Log Type="Flat File", Data column Delimeter= | , Data column Delimeter= |, Error File name="Error_rec.csv", enabled Log Row Data and Log Source Row as Yes.

                    In my error log, I am getting all error log values of  Transformation,Transformation Mapplet Name,Transformation Group,Partition Index,Transformation Row ID,Error Sequence,Error Timestamp,Error UTC Time,Error Code,Error Message,Error Type,Transformation Data,Source Mapplet Name,Source Name,Source Row ID,Source Row Type,Source Data.

                    There is no header for the error log also.

                    I just want to capture row data which was rejected by the target. I am unable to do that. I go through documents but I didn't find any way to achieve this.

                    Is there any mechanism to capture just only row data into the error log as CSV file?

                     

                    Here is a sample error log file?

                    IUpdateStrategyNode4,N/A,Output,1,1,1,09/08/2020 13:37:03,1599572223,21449,[IUpdateStrategyNode4]: REJECT: Discarding output row for transformation.

                    [error_handling_],3,""""|""100""|""abc""|""10001""|""2000004"",N/A,N/A,-1,-1,N/A

                    Expression,N/A,Output,1,7,1,09/08/2020 13:37:03,1599572223,11021,An error occurred moving data from the transformation Expression: to the transformation IUpdateStrategyNode4.,3,""1008""|""3005""|""2020-08-20 22:45:00""|""2020-08-20 22:45:00""|""""|""""|""30005"""""""""""""",N/A,N/A,-1,-1,N/A

                     

                    Note data that I want to capture is in red.

                    • 8. Re: Error handling in IICS data integration mapping task
                      Nithy S Seasoned Veteran

                      inuser522348

                       

                      Can you try without Error logging parameters in the Mapping Task? Enable Forward reject rows in the Target. It should create *output1.bad file in the */data/error/<MappingTaskFRSID> location. That should be parsable.

                       

                      Thanks!