3 Replies Latest reply on Sep 17, 2020 7:43 AM by Prithvi Chaturvedi

    mapping filed SEVERE: Spark task [InfaSpark0] failed with the following error: [com.informatica.sdk.dtm.ExecutionException: [[SPARK_1003] Spark task [InfaSpark0] failed

    VENU N New Member

      2020-09-17 03:25:09.500 <LdtmWorkflowTask-pool-5-thread-81> INFO:  Spark task is running as Yarn application with ID [application_1599667904895_60673] .

      2020-09-17 03:25:39.959 <SparkStatusCheckerMainTask> INFO: Application application_1599667904895_60673 completed with status: FAILED

      2020-09-17 03:25:39.966 <LdtmWorkflowTask-pool-5-thread-81> INFO: Deinitializing the Spark task [InfaSpark0]...

      2020-09-17 03:25:39.971 <LdtmWorkflowTask-pool-5-thread-81> INFO: DIS Custom Property: ExecutionContextOptions.DebugSparkPushdown= [null] .

      2020-09-17 03:25:40.027 <LdtmWorkflowTask-pool-5-thread-81> INFO: Deleted all configuration files.

      2020-09-17 03:25:40.033 <LdtmWorkflowTask-pool-5-thread-81> SEVERE: Spark task [InfaSpark0] failed with the following error: [com.informatica.sdk.dtm.ExecutionException: [[SPARK_1003] Spark task [InfaSpark0] failed with the following error: []. For more information about the failure, check the application log.   If the problem persists, contact Informatica Global Customer Support.]]. For more information about the failure, check the application log.   If the problem persists, contact Informatica Global Customer Support.

      2020-09-17 03:25:40.039 <LdtmWorkflowTask-pool-5-thread-81> INFO: The Integration Service started running the [Command] task [Cleanup].

      2020-09-17 03:25:40.045 <LdtmWorkflowTask-pool-5-thread-81> INFO: Created Task for [Cleanup] command [$INFA_HOME/services/shared/hadoop/CDH_5.15/scripts/HadoopFsRmRf --hadoop.home $INFA_HOME/services/shared/hadoop/CDH_5.15 --hdfsUser task-inf-cbdm-devdm --spn task-inf-cbdm-dev@BP1.AD.BP.COM --keytab /app/informatica/infa_shared/DEV/ADMIN/KEYTAB/task-inf-cbdm-dev.keytab --ccoConfPath /app/informatica/infa_shared/DEV/OP_SCORECARD/disTemp/CDL_BDM_DOM_DEV/CDL_BDM_DIS_DEV_DRS/infacdlbdmdeva/daisy_red_sql/SPARK/a4f24a0d-dee6-4cb9-832f-4d8d1e59eb08/infacco-site.xml hdfs://hanameservice//user/task-inf-cbdm-devdm/spark//sess8076760432606445618/]

      2020-09-17 03:25:40.050 <LdtmWorkflowTask-pool-5-thread-81> INFO: Created Task for [Cleanup] command [rm -rf /app/informatica/infa_shared/DEV/OP_SCORECARD/disTemp/sess8076760432606445618/]

      2020-09-17 03:25:40.055 <LdtmWorkflowTask-pool-5-thread-81> INFO: [Cleanup] [OutOfProcessTask][Cleanup ] command [rm -rf /app/informatica/infa_shared/DEV/OP_SCORECARD/disTemp/sess8076760432606445618/] is executed out of process.

      2020-09-17 03:25:40.063 <CmdExecInProcessTasks-pool-2-thread-30> INFO: [Cleanup] [HadoopFSRmRfTask]FileSystem object created from HadoopFSRmRfTask with information : DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1383674419_199732, ugi=task-inf-cbdm-devdm (auth:PROXY) via task-inf-cbdm-dev@BP1.AD.BP.COM (auth:KERBEROS)]]

      2020-09-17 03:25:40.355 <CmdExecInProcessTasks-pool-2-thread-30> WARNING: [Cleanup] [HadoopFSRmRfTask][hdfs://hanameservice//user/task-inf-cbdm-devdm/spark//sess8076760432606445618/] does not exist.

      2020-09-17 03:25:40.361 <CmdExecInProcessTasks-pool-2-thread-30> INFO: [Cleanup][$INFA_HOME/services/shared/hadoop/CDH_5.15/scripts/HadoopFsRmRf --hadoop.home $INFA_HOME/services/shared/hadoop/CDH_5.15 --hdfsUser task-inf-cbdm-devdm --spn task-inf-cbdm-dev@BP1.AD.BP.COM --keytab /app/informatica/infa_shared/DEV/ADMIN/KEYTAB/task-inf-cbdm-dev.keytab --ccoConfPath /app/informatica/infa_shared/DEV/OP_SCORECARD/disTemp/CDL_BDM_DOM_DEV/CDL_BDM_DIS_DEV_DRS/infacdlbdmdeva/daisy_red_sql/SPARK/a4f24a0d-dee6-4cb9-832f-4d8d1e59eb08/infacco-site.xml hdfs://hanameservice//user/task-inf-cbdm-devdm/spark//sess8076760432606445618/] completed

      2020-09-17 03:25:40.491 <CmdExecFCorOOPTasks-pool-3-thread-14> INFO: [Cleanup] [OutOfProcessTask]

      2020-09-17 03:25:40.496 <LdtmWorkflowTask-pool-5-thread-81> INFO: The Integration Service finished running the task [Cleanup].

      2020-09-17 03:25:40.496 <CmdExecFCorOOPTasks-pool-3-thread-14> INFO: [Cleanup][rm -rf /app/informatica/infa_shared/DEV/OP_SCORECARD/disTemp/sess8076760432606445618/] completed

      2020-09-17 03:25:40.501 <HadoopBatchDTM-pool-4-thread-1181> SEVERE: The Integration Service failed to execute the mapping.

      com.informatica.sdk.dtm.ExecutionException: [SPARK_1003] Spark task [InfaSpark0] failed with the following error: []. For more information about the failure, check the application log.   If the problem persists, contact Informatica Global Customer Support.

      at com.informatica.platform.dtm.executor.spark.monitoring.SparkStatusChecker.handleRunCompletion(SparkStatusChecker.java:410)

      at com.informatica.platform.dtm.executor.spark.monitoring.SparkStatusChecker.processRegistry(SparkStatusChecker.java:265)

      at com.informatica.platform.dtm.executor.spark.monitoring.SparkStatusChecker.run(SparkStatusChecker.java:147)

      at java.lang.Thread.run(Thread.java:748)

      2020-09-17 03:25:40.512 <HadoopBatchDTM-pool-4-thread-1181> INFO: The mapping ended with the following state: [Failed].

      2020-09-17 03:25:40.516 <HadoopBatchDTM-pool-4-thread-1181> WARNING: Number of cluster nodes used by mapping is not available.

      2020-09-17 03:25:40.521 <HadoopBatchDTM-pool-4-thread-1181> INFO: [LDTM_0075] Total time to perform the LDTM operation: 62,863 ms