5 Replies Latest reply on Feb 14, 2019 3:05 AM by user165569

    ERROR: "FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask], Hive SQL state [08S01].]" while running a mapping with Joiner transformation in Informatica BDM 10.1.1 HOTFix 1

    Tushar Borkar New Member

      Hi Team,

      I tried to load data using joiner transformation in HIVE mode and got the below error.

       

      Note : source and target both are residing on hive only.

       

      Exception Message: [[HIVE_1070] The Integration Service failed to run Hive query [exec0_query_1] for task [exec0] due to following error: Hive error code [2], Hive message [FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask], Hive SQL state [08S01].].

       

      Please let me know workaround for the same.

       

      I have added below properties in Environment variable in developer tool

       

      SET hive.exec.dynamic.partition = true;SET hive.exec.dynamic.partition.mode = nonstrict;SET hive.auto.convert.join = false;SET mapreduce.map.memory.mb=8192;SET mapreduce.reduce.memory.mb=8192;

       

       

       

      Thanks,

      Tushar

        • 1. Re: ERROR: "FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask], Hive SQL state [08S01].]" while running a mapping with Joiner transformation in Informatica BDM 10.1.1 HOTFix 1
          Swati Chadha New Member

          Hi Tushar,

           

          The error message you in the mapping log is very generic.

          Every Mapping in Hive push down mode is converted into a set of hive execution queries. From the error message shared we can see exec0_query_1 has failed. This means that the first query itself is failing.

           

          • Check if you are able to run data preview on the data sources.
          • If yes, in the mapping log you will see yarn's application ID (you will also find a link to logs).
          • Check the yarn log in the hadoop cluster to find what is going wrong.

           

          Regards,
          Swati

          • 2. Re: ERROR: "FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask], Hive SQL state [08S01].]" while running a mapping with Joiner transformation in Informatica BDM 10.1.1 HOTFix 1
            Tushar Borkar New Member

            Hi Swati,

             

            Thanks for your reply. I have followed all the steps which you mentioned and got the below error.

             

            Error : Java Heap Space.

             

            Please let me know the workaround for the same.

             

            Thanks,

            Tushar

            09860337350

            • 3. Re: ERROR: "FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask], Hive SQL state [08S01].]" while running a mapping with Joiner transformation in Informatica BDM 10.1.1 HOTFix 1
              Swati Chadha New Member

              Hi Tushar,

               

              Again it will depend on which java process is getting the OutOfMemoryError error.

               

              Can you please paste the compete error message ?

               

              Regards,

              Swati

              • 4. Re: ERROR: "FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask], Hive SQL state [08S01].]" while running a mapping with Joiner transformation in Informatica BDM 10.1.1 HOTFix 1
                Tushar Borkar New Member

                Please see below complete error -

                 

                Log Type: syslog

                Log Upload Time: Mon Apr 23 11:49:27 +0000 2018

                Log Length: 50144

                2018-04-23 11:46:18,844 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1523451994798_0239_000001
                2018-04-23 11:46:19,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
                2018-04-23 11:46:19,160 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@6be968ce)
                2018-04-23 11:46:19,344 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.11.13:8020, Ident: (token for infa: HDFS_DELEGATION_TOKEN owner=infa, renewer=yarn, realUser=clouderainfaqa@SGWS.LOCAL, issueDate=1524483973661, maxDate=1525088773661, sequenceNumber=27467, masterKeyId=80)
                2018-04-23 11:46:19,345 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: HIVE_DELEGATION_TOKEN, Service: InfaImpersonationToken, Ident: 00 04 69 6e 66 61 04 69 6e 66 61 19 63 6c 6f 75 64 65 72 61 69 6e 66 61 71 61 40 53 47 57 53 2e 4c 4f 43 41 4c 8a 01 62 f2 53 61 58 8a 01 63 16 5f e5 58 8e 01 be 07
                2018-04-23 11:46:19,970 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
                2018-04-23 11:46:20,087 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
                2018-04-23 11:46:20,089 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
                2018-04-23 11:46:20,153 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
                2018-04-23 11:46:20,154 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
                2018-04-23 11:46:20,155 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
                2018-04-23 11:46:20,156 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
                2018-04-23 11:46:20,156 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
                2018-04-23 11:46:20,163 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
                2018-04-23 11:46:20,163 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
                2018-04-23 11:46:20,164 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
                2018-04-23 11:46:20,224 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://sowueswtauclm01.sgws.local:8020]
                2018-04-23 11:46:20,264 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://sowueswtauclm01.sgws.local:8020]
                2018-04-23 11:46:20,297 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://sowueswtauclm01.sgws.local:8020]
                2018-04-23 11:46:20,307 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
                2018-04-23 11:46:20,347 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
                2018-04-23 11:46:20,563 WARN [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: org.apache.commons.configuration.ConfigurationException: Unable to load the configuration from the URL file:/run/cloudera-scm-agent/process/523-yarn-NODEMANAGER/hadoop-metrics2.properties
                2018-04-23 11:46:20,615 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1523451994798_0239 to jobTokenSecretManager
                2018-04-23 11:46:20,738 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1523451994798_0239 because: not enabled; too much input; too much RAM;
                2018-04-23 11:46:20,769 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1523451994798_0239 = 269526605. Number of splits = 1
                2018-04-23 11:46:20,769 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1523451994798_0239 = 0
                2018-04-23 11:46:20,769 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1523451994798_0239Job Transitioned from NEW to INITED
                2018-04-23 11:46:20,771 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1523451994798_0239.
                2018-04-23 11:46:20,804 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
                2018-04-23 11:46:20,813 INFO [Socket Reader #1 for port 36873] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36873
                2018-04-23 11:46:20,831 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
                2018-04-23 11:46:20,867 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
                2018-04-23 11:46:20,867 INFO [IPC Server listener on 36873] org.apache.hadoop.ipc.Server: IPC Server listener on 36873: starting
                2018-04-23 11:46:20,868 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at sowueswtaucld02.sgws.local/10.70.11.78:36873
                2018-04-23 11:46:20,924 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
                2018-04-23 11:46:20,929 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
                2018-04-23 11:46:20,933 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
                2018-04-23 11:46:20,942 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
                2018-04-23 11:46:20,946 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
                2018-04-23 11:46:20,946 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
                2018-04-23 11:46:20,949 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
                2018-04-23 11:46:20,949 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
                2018-04-23 11:46:20,959 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 43705
                2018-04-23 11:46:20,959 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
                2018-04-23 11:46:20,992 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/hadoop-yarn-common-2.6.0-cdh5.10.1.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_43705_mapreduce____douhj0/webapp
                2018-04-23 11:46:21,317 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:43705
                2018-04-23 11:46:21,319 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 43705
                2018-04-23 11:46:21,842 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
                2018-04-23 11:46:21,845 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1523451994798_0239
                2018-04-23 11:46:21,847 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
                2018-04-23 11:46:21,848 INFO [Socket Reader #1 for port 33539] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33539
                2018-04-23 11:46:21,901 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
                2018-04-23 11:46:21,901 INFO [IPC Server listener on 33539] org.apache.hadoop.ipc.Server: IPC Server listener on 33539: starting
                2018-04-23 11:46:21,963 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
                2018-04-23 11:46:21,963 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
                2018-04-23 11:46:21,964 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
                2018-04-23 11:46:22,069 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at sowueswtauclm01.sgws.local/10.70.11.13:8030
                2018-04-23 11:46:22,144 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:39195, vCores:8>
                2018-04-23 11:46:22,144 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.users.infa
                2018-04-23 11:46:22,149 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
                2018-04-23 11:46:22,149 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
                2018-04-23 11:46:22,165 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1523451994798_0239Job Transitioned from INITED to SETUP
                2018-04-23 11:46:22,167 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
                2018-04-23 11:46:22,169 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1523451994798_0239Job Transitioned from SETUP to RUNNING
                2018-04-23 11:46:22,258 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1523451994798_0239, File: hdfs://sowueswtauclm01.sgws.local:8020/user/infa/.staging/job_1523451994798_0239/job_1523451994798_0239_1.jhist
                2018-04-23 11:46:22,417 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for infa (auth:SIMPLE)
                2018-04-23 11:46:22,418 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1523451994798_0239_m_000000 Task Transitioned from NEW to SCHEDULED
                2018-04-23 11:46:22,431 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
                2018-04-23 11:46:22,439 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:2048, vCores:1>
                2018-04-23 11:46:22,446 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for infa (auth:TOKEN) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB
                2018-04-23 11:46:22,595 INFO [IPC Server handler 0 on 36873] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for MAP   job_1523451994798_0239. Report-size will be 1
                2018-04-23 11:46:22,648 INFO [IPC Server handler 0 on 36873] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for REDUCE   job_1523451994798_0239. Report-size will be 0
                2018-04-23 11:46:22,679 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://sowueswtauclm01.sgws.local:8020]
                2018-04-23 11:46:23,149 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
                2018-04-23 11:46:23,176 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1523451994798_0239: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:56745, vCores:14> knownNMs=3
                2018-04-23 11:46:24,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
                2018-04-23 11:46:24,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1523451994798_0239_01_000002 to attempt_1523451994798_0239_m_000000_0
                2018-04-23 11:46:24,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
                2018-04-23 11:46:24,313 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://sowueswtauclm01.sgws.local:8020/user/infa/.staging/job_1523451994798_0239/job.jar
                2018-04-23 11:46:24,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/infa/.staging/job_1523451994798_0239/job.xml
                2018-04-23 11:46:24,329 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #2 tokens and #1 secret keys for NM use for launching container
                2018-04-23 11:46:24,330 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 3
                2018-04-23 11:46:24,330 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
                2018-04-23 11:46:24,379 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
                2018-04-23 11:46:24,382 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1523451994798_0239_01_000002 taskAttempt attempt_1523451994798_0239_m_000000_0
                2018-04-23 11:46:24,383 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1523451994798_0239_m_000000_0
                2018-04-23 11:46:24,433 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1523451994798_0239_m_000000_0 : 13562
                2018-04-23 11:46:24,434 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1523451994798_0239_m_000000_0] using containerId: [container_1523451994798_0239_01_000002 on NM: [sowueswtaucld02.sgws.local:8041]
                2018-04-23 11:46:24,437 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
                2018-04-23 11:46:24,437 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1523451994798_0239_m_000000 Task Transitioned from SCHEDULED to RUNNING
                2018-04-23 11:46:24,700 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for clouderainfaqa@SGWS.LOCAL (auth:SIMPLE)
                2018-04-23 11:46:24,702 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for clouderainfaqa@SGWS.LOCAL (auth:TOKEN) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB
                2018-04-23 11:46:25,260 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1523451994798_0239: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:52649, vCores:12> knownNMs=3
                2018-04-23 11:46:26,587 INFO [Socket Reader #1 for port 33539] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1523451994798_0239 (auth:SIMPLE)
                2018-04-23 11:46:26,628 INFO [Socket Reader #1 for port 33539] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1523451994798_0239 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
                2018-04-23 11:46:26,637 INFO [IPC Server handler 1 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1523451994798_0239_m_000002 asked for a task
                2018-04-23 11:46:26,637 INFO [IPC Server handler 1 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1523451994798_0239_m_000002 given task: attempt_1523451994798_0239_m_000000_0
                2018-04-23 11:46:40,002 INFO [IPC Server handler 5 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_0 is : 0.0
                2018-04-23 11:46:46,033 INFO [IPC Server handler 3 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_0 is : 0.0
                2018-04-23 11:46:52,279 INFO [IPC Server handler 6 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_0 is : 0.0
                2018-04-23 11:46:58,578 INFO [IPC Server handler 8 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_0 is : 0.0
                2018-04-23 11:47:04,703 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for clouderainfaqa@SGWS.LOCAL (auth:SIMPLE)
                2018-04-23 11:47:04,705 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for clouderainfaqa@SGWS.LOCAL (auth:TOKEN) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB
                2018-04-23 11:47:05,696 INFO [IPC Server handler 7 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_0 is : 0.0
                2018-04-23 11:47:12,852 INFO [IPC Server handler 2 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_0 is : 0.0
                2018-04-23 11:47:19,223 FATAL [IPC Server handler 6 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1523451994798_0239_m_000000_0 - exited : Java heap space
                2018-04-23 11:47:19,223 INFO [IPC Server handler 6 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_0: Error: Java heap space
                2018-04-23 11:47:19,225 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_0: Error: Java heap space
                2018-04-23 11:47:19,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_FINISHING_CONTAINER
                2018-04-23 11:47:19,238 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sowueswtaucld02.sgws.local
                2018-04-23 11:47:19,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
                2018-04-23 11:47:19,240 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1523451994798_0239_m_000000_1 to list of failed maps
                2018-04-23 11:47:19,339 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
                2018-04-23 11:47:19,341 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1523451994798_0239: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:55721, vCores:14> knownNMs=3
                2018-04-23 11:47:20,346 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1523451994798_0239_01_000002
                2018-04-23 11:47:20,346 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
                2018-04-23 11:47:20,346 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_0:
                2018-04-23 11:47:20,346 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1523451994798_0239_01_000003, NodeId: sowueswtaucld02.sgws.local:8041, NodeHttpAddress: sowueswtaucld02.sgws.local:8042, Resource: <memory:2048, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.70.11.78:8041 }, ] to fast fail map
                2018-04-23 11:47:20,346 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
                2018-04-23 11:47:20,346 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_0 TaskAttempt Transitioned from FAIL_FINISHING_CONTAINER to FAILED
                2018-04-23 11:47:20,347 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1523451994798_0239_01_000003 to attempt_1523451994798_0239_m_000000_1
                2018-04-23 11:47:20,347 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:0 RackLocal:1
                2018-04-23 11:47:20,347 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1523451994798_0239_01_000002 taskAttempt attempt_1523451994798_0239_m_000000_0
                2018-04-23 11:47:20,349 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
                2018-04-23 11:47:20,349 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1523451994798_0239_01_000003 taskAttempt attempt_1523451994798_0239_m_000000_1
                2018-04-23 11:47:20,349 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1523451994798_0239_m_000000_1
                2018-04-23 11:47:20,357 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1523451994798_0239_m_000000_1 : 13562
                2018-04-23 11:47:20,357 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1523451994798_0239_m_000000_1] using containerId: [container_1523451994798_0239_01_000003 on NM: [sowueswtaucld02.sgws.local:8041]
                2018-04-23 11:47:20,358 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
                2018-04-23 11:47:21,348 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1523451994798_0239: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:55721, vCores:14> knownNMs=3
                2018-04-23 11:47:21,876 INFO [Socket Reader #1 for port 33539] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1523451994798_0239 (auth:SIMPLE)
                2018-04-23 11:47:21,886 INFO [Socket Reader #1 for port 33539] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1523451994798_0239 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
                2018-04-23 11:47:21,886 INFO [IPC Server handler 10 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1523451994798_0239_m_000003 asked for a task
                2018-04-23 11:47:21,886 INFO [IPC Server handler 10 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1523451994798_0239_m_000003 given task: attempt_1523451994798_0239_m_000000_1
                2018-04-23 11:47:34,813 INFO [IPC Server handler 2 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_1 is : 0.0
                2018-04-23 11:47:40,874 INFO [IPC Server handler 10 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_1 is : 0.0
                2018-04-23 11:47:47,355 INFO [IPC Server handler 8 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_1 is : 0.0
                2018-04-23 11:47:52,363 FATAL [IPC Server handler 7 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1523451994798_0239_m_000000_1 - exited : Java heap space
                2018-04-23 11:47:52,363 INFO [IPC Server handler 7 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_1: Error: Java heap space
                2018-04-23 11:47:52,364 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_1: Error: Java heap space
                2018-04-23 11:47:52,365 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_1 TaskAttempt Transitioned from RUNNING to FAIL_FINISHING_CONTAINER
                2018-04-23 11:47:52,366 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node sowueswtaucld02.sgws.local
                2018-04-23 11:47:52,366 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_2 TaskAttempt Transitioned from NEW to UNASSIGNED
                2018-04-23 11:47:52,366 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1523451994798_0239_m_000000_2 to list of failed maps
                2018-04-23 11:47:52,387 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:0 RackLocal:1
                2018-04-23 11:47:52,389 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1523451994798_0239: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:55721, vCores:14> knownNMs=3
                2018-04-23 11:47:53,390 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1523451994798_0239_01_000003
                2018-04-23 11:47:53,391 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
                2018-04-23 11:47:53,391 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_1:
                2018-04-23 11:47:53,391 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1523451994798_0239_01_000004, NodeId: sowueswtaucld03.sgws.local:8041, NodeHttpAddress: sowueswtaucld03.sgws.local:8042, Resource: <memory:2048, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.70.11.116:8041 }, ] to fast fail map
                2018-04-23 11:47:53,391 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
                2018-04-23 11:47:53,391 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_1 TaskAttempt Transitioned from FAIL_FINISHING_CONTAINER to FAILED
                2018-04-23 11:47:53,391 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1523451994798_0239_01_000004 to attempt_1523451994798_0239_m_000000_2
                2018-04-23 11:47:53,391 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:0 HostLocal:0 RackLocal:1
                2018-04-23 11:47:53,391 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1523451994798_0239_01_000003 taskAttempt attempt_1523451994798_0239_m_000000_1
                2018-04-23 11:47:53,393 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
                2018-04-23 11:47:53,393 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1523451994798_0239_01_000004 taskAttempt attempt_1523451994798_0239_m_000000_2
                2018-04-23 11:47:53,393 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1523451994798_0239_m_000000_2
                2018-04-23 11:47:53,401 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1523451994798_0239_m_000000_2 : 13562
                2018-04-23 11:47:53,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1523451994798_0239_m_000000_2] using containerId: [container_1523451994798_0239_01_000004 on NM: [sowueswtaucld03.sgws.local:8041]
                2018-04-23 11:47:53,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
                2018-04-23 11:47:54,392 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1523451994798_0239: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:55721, vCores:14> knownNMs=3
                2018-04-23 11:47:57,376 INFO [Socket Reader #1 for port 33539] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1523451994798_0239 (auth:SIMPLE)
                2018-04-23 11:47:57,387 INFO [Socket Reader #1 for port 33539] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1523451994798_0239 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
                2018-04-23 11:47:57,387 INFO [IPC Server handler 2 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1523451994798_0239_m_000004 asked for a task
                2018-04-23 11:47:57,388 INFO [IPC Server handler 2 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1523451994798_0239_m_000004 given task: attempt_1523451994798_0239_m_000000_2
                2018-04-23 11:48:04,698 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for clouderainfaqa@SGWS.LOCAL (auth:SIMPLE)
                2018-04-23 11:48:04,700 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for clouderainfaqa@SGWS.LOCAL (auth:TOKEN) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB
                2018-04-23 11:48:10,093 INFO [IPC Server handler 8 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_2 is : 0.0
                2018-04-23 11:48:16,276 INFO [IPC Server handler 7 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_2 is : 0.0
                2018-04-23 11:48:22,525 INFO [IPC Server handler 10 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_2 is : 0.0
                2018-04-23 11:48:27,737 FATAL [IPC Server handler 4 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1523451994798_0239_m_000000_2 - exited : Java heap space
                2018-04-23 11:48:27,737 INFO [IPC Server handler 4 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_2: Error: Java heap space
                2018-04-23 11:48:27,738 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_2: Error: Java heap space
                2018-04-23 11:48:27,739 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_2 TaskAttempt Transitioned from RUNNING to FAIL_FINISHING_CONTAINER
                2018-04-23 11:48:27,739 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sowueswtaucld03.sgws.local
                2018-04-23 11:48:27,739 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_3 TaskAttempt Transitioned from NEW to UNASSIGNED
                2018-04-23 11:48:27,740 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1523451994798_0239_m_000000_3 to list of failed maps
                2018-04-23 11:48:28,438 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:0 HostLocal:0 RackLocal:1
                2018-04-23 11:48:28,439 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1523451994798_0239: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:55721, vCores:14> knownNMs=3
                2018-04-23 11:48:29,441 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1523451994798_0239_01_000004
                2018-04-23 11:48:29,441 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
                2018-04-23 11:48:29,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_2:
                2018-04-23 11:48:29,441 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1523451994798_0239_01_000005, NodeId: sowueswtaucld03.sgws.local:8041, NodeHttpAddress: sowueswtaucld03.sgws.local:8042, Resource: <memory:2048, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.70.11.116:8041 }, ] to fast fail map
                2018-04-23 11:48:29,441 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
                2018-04-23 11:48:29,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_2 TaskAttempt Transitioned from FAIL_FINISHING_CONTAINER to FAILED
                2018-04-23 11:48:29,442 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1523451994798_0239_01_000005 to attempt_1523451994798_0239_m_000000_3
                2018-04-23 11:48:29,442 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:0 RackLocal:1
                2018-04-23 11:48:29,442 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1523451994798_0239_01_000004 taskAttempt attempt_1523451994798_0239_m_000000_2
                2018-04-23 11:48:29,443 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
                2018-04-23 11:48:29,444 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1523451994798_0239_01_000005 taskAttempt attempt_1523451994798_0239_m_000000_3
                2018-04-23 11:48:29,444 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1523451994798_0239_m_000000_3
                2018-04-23 11:48:29,452 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1523451994798_0239_m_000000_3 : 13562
                2018-04-23 11:48:29,452 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1523451994798_0239_m_000000_3] using containerId: [container_1523451994798_0239_01_000005 on NM: [sowueswtaucld03.sgws.local:8041]
                2018-04-23 11:48:29,452 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_3 TaskAttempt Transitioned from ASSIGNED to RUNNING
                2018-04-23 11:48:30,443 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1523451994798_0239: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:55721, vCores:14> knownNMs=3
                2018-04-23 11:48:30,990 INFO [Socket Reader #1 for port 33539] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1523451994798_0239 (auth:SIMPLE)
                2018-04-23 11:48:31,000 INFO [Socket Reader #1 for port 33539] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1523451994798_0239 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
                2018-04-23 11:48:31,000 INFO [IPC Server handler 5 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1523451994798_0239_m_000005 asked for a task
                2018-04-23 11:48:31,000 INFO [IPC Server handler 5 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1523451994798_0239_m_000005 given task: attempt_1523451994798_0239_m_000000_3
                2018-04-23 11:48:43,737 INFO [IPC Server handler 9 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_3 is : 0.0
                2018-04-23 11:48:49,910 INFO [IPC Server handler 11 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_3 is : 0.0
                2018-04-23 11:48:56,136 INFO [IPC Server handler 7 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_3 is : 0.0
                2018-04-23 11:49:02,706 INFO [IPC Server handler 4 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_3 is : 0.0
                2018-04-23 11:49:04,701 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for clouderainfaqa@SGWS.LOCAL (auth:SIMPLE)
                2018-04-23 11:49:04,702 INFO [Socket Reader #1 for port 36873] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for clouderainfaqa@SGWS.LOCAL (auth:TOKEN) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB
                2018-04-23 11:49:08,921 INFO [IPC Server handler 12 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1523451994798_0239_m_000000_3 is : 0.0
                2018-04-23 11:49:20,196 FATAL [IPC Server handler 2 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1523451994798_0239_m_000000_3 - exited : Java heap space
                2018-04-23 11:49:20,196 INFO [IPC Server handler 2 on 33539] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_3: Error: Java heap space
                2018-04-23 11:49:20,197 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1523451994798_0239_m_000000_3: Error: Java heap space
                2018-04-23 11:49:20,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_3 TaskAttempt Transitioned from RUNNING to FAIL_FINISHING_CONTAINER
                2018-04-23 11:49:20,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1523451994798_0239_m_000000 Task Transitioned from RUNNING to FAILED
                2018-04-23 11:49:20,200 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node sowueswtaucld03.sgws.local
                2018-04-23 11:49:20,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
                2018-04-23 11:49:20,201 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0
                2018-04-23 11:49:20,201 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1523451994798_0239Job Transitioned from RUNNING to FAIL_ABORT
                2018-04-23 11:49:20,202 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
                2018-04-23 11:49:20,207 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1523451994798_0239Job Transitioned from FAIL_ABORT to FAILED
                2018-04-23 11:49:20,208 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
                2018-04-23 11:49:20,209 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
                2018-04-23 11:49:20,209 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
                2018-04-23 11:49:20,209 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
                2018-04-23 11:49:20,209 INFO [Thread-75] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
                2018-04-23 11:49:20,209 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
                2018-04-23 11:49:20,209 INFO [Thread-75] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
                2018-04-23 11:49:20,245 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://sowueswtauclm01.sgws.local:8020/user/infa/.staging/job_1523451994798_0239/job_1523451994798_0239_1.jhist to hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239-1524483974039-infa-INSERT+INTO+TABLE+SAL..._sales_stg_tgt.col99%28Stage-1524484160201-0-0-FAILED-root.users.infa-1524483982160.jhist_tmp
                2018-04-23 11:49:20,270 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239-1524483974039-infa-INSERT+INTO+TABLE+SAL..._sales_stg_tgt.col99%28Stage-1524484160201-0-0-FAILED-root.users.infa-1524483982160.jhist_tmp
                2018-04-23 11:49:20,272 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://sowueswtauclm01.sgws.local:8020/user/infa/.staging/job_1523451994798_0239/job_1523451994798_0239_1_conf.xml to hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239_conf.xml_tmp
                2018-04-23 11:49:20,298 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239_conf.xml_tmp
                2018-04-23 11:49:20,304 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239.summary_tmp to hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239.summary
                2018-04-23 11:49:20,305 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239_conf.xml_tmp to hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239_conf.xml
                2018-04-23 11:49:20,307 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239-1524483974039-infa-INSERT+INTO+TABLE+SAL..._sales_stg_tgt.col99%28Stage-1524484160201-0-0-FAILED-root.users.infa-1524483982160.jhist_tmp to hdfs://sowueswtauclm01.sgws.local:8020/user/history/done_intermediate/infa/job_1523451994798_0239-1524483974039-infa-INSERT+INTO+TABLE+SAL..._sales_stg_tgt.col99%28Stage-1524484160201-0-0-FAILED-root.users.infa-1524483982160.jhist
                2018-04-23 11:49:20,308 INFO [Thread-75] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
                2018-04-23 11:49:20,308 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1523451994798_0239_m_000000_3
                2018-04-23 11:49:20,321 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1523451994798_0239_m_000000_3 TaskAttempt Transitioned from FAIL_FINISHING_CONTAINER to FAILED
                2018-04-23 11:49:20,323 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Task failed task_1523451994798_0239_m_000000
                Job failed as tasks failed. failedMaps:1 failedReduces:0

                2018-04-23 11:49:20,323 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://sowueswtauclm01.sgws.local:19888/jobhistory/job/job_1523451994798_0239
                2018-04-23 11:49:20,328 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
                2018-04-23 11:49:21,330 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:0 RackLocal:1
                2018-04-23 11:49:21,331 INFO [Thread-75] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://sowueswtauclm01.sgws.local:8020 /user/infa/.staging/job_1523451994798_0239
                2018-04-23 11:49:21,338 INFO [Thread-75] org.apache.hadoop.ipc.Server: Stopping server on 33539
                2018-04-23 11:49:21,342 INFO [IPC Server listener on 33539] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 33539
                2018-04-23 11:49:21,342 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
                2018-04-23 11:49:21,342 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
                2018-04-23 11:49:21,343 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
                2018-04-23 11:49:26,343 INFO [Thread-75] org.apache.hadoop.ipc.Server: Stopping server on 36873
                2018-04-23 11:49:26,343 INFO [IPC Server listener on 36873] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 36873
                2018-04-23 11:49:26,343 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
                2018-04-23 11:49:26,345 INFO [Thread-75] org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0

                • 5. Re: ERROR: "FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask], Hive SQL state [08S01].]" while running a mapping with Joiner transformation in Informatica BDM 10.1.1 HOTFix 1
                  user165569 Guru

                  Hi Tushar,

                   

                  For 10.1.1 HF1 version, you can increase the Xmx value for MapReduce job using infapdo.java.opts in the hadoopEnv.properties file.

                   

                  Thanks,
                  Ninju

                  Informatica Support