5 Replies Latest reply on Dec 20, 2018 8:09 PM by Rajan Rath

    Why the Integration service goes down frequently?

    Vidya M C New Member

      Hi,

       

      Powercenter Integration service goes down frequently. and below is the error message I get every time.

       

      Same error was available in node.log as well.

       

       

       

       

      Powercenter GRID DB has no issue in the as per the DB logs . And the Integration service is hosted on the cloud server(unix) and there was no issue found on the server as well.

       

      (Once the memory usage in the server had reached 98% - can that be the reason for powercenter Integration service to go down?)

       

      May I know the reason why it goes down frequently?

       

      Or where I have to check to get the exact reason for Integration service going down?

        • 1. Re: Why the Integration service goes down frequently?
          Rajan Rath Guru

          Hello Vidya,

           

          The log (highlighted) is part of the Domain log I believe. You will need to check the Integration Service log and the Repository Service log for more details on the Integration Service failure.

          It could be a possibility that the Integration Service is either terminating or crashing. In such cases you can verify the $INFA_HOME/server/bin directory and check if there are any core files generated.

           

          The other possibilities could be that the Informatica node is going down causing a failover of the services to the other nodes or the Repository Service is going down and hence bringing down the Integration Service.

          There could be ample of reasons for such issues and this needs to be reviewed one after the other based on the symptom.

           

          Hope this helps.

           

          Regards,

          Rajan

          • 2. Re: Why the Integration service goes down frequently?
            Nico Heinze Guru

            Network communication has constantly increased in complexity as well as in sheer mass over the years. That's the (main) reason why Informatica has introduced the resilience timeout values in the Informatica domain.

            Please try increasing all related resilience timeout values (in the domain, for the integration and repository services, and for all underlying DB connections) e.g. from the default 180 seconds to 600 seconds. That might help.

             

            In general my suggestion in such cases is to open a service request at Informatica Global Customer Support. Due to the ever-increasing complexity of networks it can take weeks or even months to identify the root cause of such issues, and from my own time at Informatica I recall that almost every customer setup needs individual steps to identify the root causes of such troubles. This is combersome and will probably take some time.

             

            Regards,

            Nico

            1 of 1 people found this helpful
            • 3. Re: Why the Integration service goes down frequently?
              Vidya M C New Member

              Hi Rajan and Nico

               

              Thanks for your responses.

               

              @Rajan rath - I checked the mentioned directory , $INFA_HOME/server/bin - there are no core files generated here .

               

              @Nico - As I don't have the Global support creds for now. So It will take little long to raise a request and get the response from them

               

               

              I have the below logs , Will this help to identify the issue. and every day the service goes down at the same time i.e around 10 PM CET

               

              catalina.out

               

              [2018-Dec-12 22:00:56.400]PCSF Thread Group handleUnhandledException

              Internal error. An exception occurred in a PCSF or PCSF base servlet threads. See the following stacktrace. Contact Informatica Global Customer Support.

              Thread Name: [Monitor Guaranteed Message Logger Thread]

                      at org.beepcore.beep.core.Frame.parseHeader(Frame.java:299)

                      at org.beepcore.beep.core.SessionImpl.createFrame(SessionImpl.java:872)

              Exception class: [java.lang.OutOfMemoryError]

                      at org.beepcore.beep.transport.tcp.TCPSession.processCoreFrame(TCPSession.java:581)

              Exception message: [Java heap space]

              Stack trace:

              Exception in thread "pool-1-thread-156"         at org.beepcore.beep.transport.tcp.TCPSession.processNextFrame(TCPSession.java:481)

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession.processNextFrame(TCPNonBlockingSession.java:356)

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession$1.run(TCPNonBlockingSession.java:253)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

                      at java.lang.Thread.run(Thread.java:662)

              [2018-Dec-12 22:00:56.401]PCSF Thread Group handleUnhandledException

              Internal error. An exception occurred in a PCSF or PCSF base servlet threads. See the following stacktrace. Contact Informatica Global Customer Support.

              Thread Name: [Master Elect Data Writer]

              Exception class: [java.lang.OutOfMemoryError]

              Exception message: [Java heap space]

              Stack trace:

              com.informatica.jdbc.oracle.OracleImplConnection.a(Unknown Source)

              com.informatica.jdbc.oraclebase.dde3.p(Unknown Source)

              com.informatica.jdbc.oraclebase.dde3.a(Unknown Source)

              com.informatica.jdbc.oraclebase.dddz.a(Unknown Source)

              com.informatica.jdbc.oraclebase.ddt.a(Unknown Source)

              com.informatica.jdbc.oraclebase.BaseConnection.prepareStatement(Unknown Source)

              com.informatica.jdbc.oraclebase.BaseConnection.prepareStatement(Unknown Source)

              com.informatica.isp.domainservice.MasterElector.updateMyRowInDB(MasterElector.java:854)

              com.informatica.isp.domainservice.MasterElector.access$7(MasterElector.java:851)

              com.informatica.isp.domainservice.MasterElector$MasterElectDataWriter.updateRow(MasterElector.java:1013)

              com.informatica.isp.domainservice.MasterElector$MasterElectDataWriter.run(MasterElector.java:940)

              java.lang.Thread.run(Thread.java:662)

              Exception in thread "pool-1-thread-151" java.lang.OutOfMemoryError: Java heap space

                      at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)

                      at java.lang.StringBuilder.<init>(StringBuilder.java:68)

                      at java.net.Inet4Address.numericToTextFormat(Inet4Address.java:356)

                      at java.net.Inet4Address.getHostAddress(Inet4Address.java:311)

                      at java.net.InetAddress.toString(InetAddress.java:661)

                      at java.lang.String.valueOf(String.java:2826)

                      at java.lang.StringBuilder.append(StringBuilder.java:115)

                      at sun.nio.ch.SocketAdaptor.toString(SocketAdaptor.java:446)

                      at org.beepcore.beep.transport.tcp.TCPSession.processCoreFrame(TCPSession.java:644)

                      at org.beepcore.beep.transport.tcp.TCPSession.processNextFrame(TCPSession.java:481)

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession.processNextFrame(TCPNonBlockingSession.java:356)

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession$1.run(TCPNonBlockingSession.java:253)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

                      at java.lang.Thread.run(Thread.java:662)

              java.lang.OutOfMemoryError: Java heap space

              java.lang.OutOfMemoryError: Java heap space

                      at org.beepcore.beep.transport.tcp.TCPSession.processCoreFrame(TCPSession.java:582)

                      at org.beepcore.beep.transport.tcp.TCPSession.processNextFrame(TCPSession.java:481)

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession.processNextFrame(TCPNonBlockingSession.java:356)

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession$1.run(TCPNonBlockingSession.java:253)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

                      at java.lang.Thread.run(Thread.java:662)

              An exception occurred when running a PowerCenter application service. See the following messages for a stack trace.

              Thread Name: catalina-exec-710

              Exception class: java.lang.OutOfMemoryError

              Exception message: Java heap space

              Stack trace:

              The Service Manager is stopping the Tomcat process.

              [Wed Dec 12 22:01:01 CET 2018] Aborting the domain service.

              [2018-Dec-12 22:01:01.495] LogServiceAgent deinit method is called.

              [2018-Dec-12 22:01:02.495] Plugin Registry service deinit method is called.

              [2018-Dec-12 22:01:02.496] Monitoring service deinit method is called.

              [2018-Dec-12 22:01:02.548] Licensing service deinit method is called.

              [2018-Dec-12 22:01:02.548] Called the alert domain function deinit method.

              [2018-Dec-12 22:01:02.548] Edr service deinit method is called.

              [2018-Dec-12 22:01:02.549] User Management service deinit method is called.

              [2018-Dec-12 22:01:02.549] Log service deinit method is called.

              [2018-Dec-12 22:01:02.549] Domain Configuration service deinit method is called.

              [2018-Dec-12 22:02:58.757] Domain service deinit method is called. Aborting the domain service.

              2018-Dec-12 23:28:08.240 Starting Tomcat on HTTP port [7005].

               

               

              node.log

               

              2018-12-12 22:01:01,401 ERROR [Domain Monitor] [DOM_10095] The master gateway data in the domain configuration repository was updated during the last refresh time interval 31270.

              2018-12-12 22:01:01,404 FATAL [Domain Monitor] [DOM_10094] Cannot update the data for the master gateway node [vm00000292_PR_951] within the refresh interval time [360000]. The node will not continue as a master gateway node. Verify that the connection to the domain configuration repository database is valid.

              2018-12-12 22:01:01,606 ERROR [Monitor Guaranteed Message File Thread] [CCM_10342] The following exception occurred: [null].

              java.util.concurrent.RejectedExecutionException: null

               

              node_jsf.log

               

              2018-12-12 22:01:06,626 ERROR [Thread-73][com.informatica.pcsf.servicesframework.client.impl.ClientFactoryImpl] [DTF_0001] An error occurred in the underlying transport layer: [[sendMSG]: Channel is in [7] state.]

              com.informatica.pcsf.datatransport.DataTransportException: [DTF_0001] An error occurred in the underlying transport layer: [[sendMSG]: Channel is in [7] state.]

                      at com.informatica.pcsf.datatransport.impl.DataTransportChannelImpl.sendRequest(DataTransportChannelImpl.java:73)

                      at com.informatica.pcsf.servicesframework.client.impl.ClientFactoryImpl.sendHeartbeatToRecipient(ClientFactoryImpl.java:603)

                      at com.informatica.pcsf.servicesframework.client.impl.ClientFactoryImpl.isAlive(ClientFactoryImpl.java:500)

                      at com.informatica.isp.monitoringservice.query.StatsMRSServiceConnectionPool$NotificationConnectionMonitorThread.run(StatsMRSServiceConnectionPool.java:571)

                      at java.lang.Thread.run(Thread.java:662)

              Caused by: org.beepcore.beep.core.BEEPException: [sendMSG]: Channel is in [7] state.

                      at org.beepcore.beep.core.ChannelImpl.sendMsgInternal(ChannelImpl.java:399)

                      at org.beepcore.beep.core.ChannelImpl.sendMSG(ChannelImpl.java:446)

                      at com.informatica.pcsf.datatransport.impl.DataTransportChannelImpl.sendRequest(DataTransportChannelImpl.java:71)

                      ... 4 more

              2018-12-12 22:01:06,630 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:01:16,630 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:01:26,631 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:01:28,752 ERROR [localhost-startStop-2][org.beepcore.beep.transport.tcp.TCPNonBlockingSession] [BeepCore_0021] Error channel state [2]

              2018-12-12 22:01:36,631 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:01:46,632 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:01:56,632 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:01:58,754 ERROR [localhost-startStop-2][org.beepcore.beep.transport.tcp.TCPNonBlockingSession] [BeepCore_0021] Error channel state [2]

              2018-12-12 22:02:06,633 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:02:16,633 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:02:26,633 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:02:28,755 ERROR [localhost-startStop-2][org.beepcore.beep.transport.tcp.TCPNonBlockingSession] [BeepCore_0021] Error channel state [2]

              2018-12-12 22:02:36,634 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:02:46,634 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:02:56,635 FATAL [Thread-73][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0003] Cannot create a new session while the DataTransportFramework is shutting down.

              2018-12-12 22:02:58,756 ERROR [localhost-startStop-2][org.beepcore.beep.transport.tcp.TCPNonBlockingSession] [BeepCore_0021] Error channel state [2]

              2018-12-13 01:00:00,255 ERROR [Timer-0][com.informatica.pcsf.datatransport.impl.DataTransportFrameworkImpl] [DTF_0032] Service [DomainService] does not support secure communication connection that the client requested.

              • 4. Re: Why the Integration service goes down frequently?
                Nico Heinze Guru

                Not 100% sure; to me this sounds as if your system has some connectivity issue at app. 10 p.m. each day.

                Is this the time when the repository DB undergoes a backup?

                Or that the system itself is backed up?

                That would explain this story.

                 

                Regards,

                Nico

                • 5. Re: Why the Integration service goes down frequently?
                  Rajan Rath Guru

                  Hello Vidya,

                   

                  The Informatica services goes down with a Java OutOfMemoryError (OOME).

                  What is the maximum heap size value set to? You may try increasing the Xmx value for the Informatica services to avoid any crash.

                   

                  The Xmx value can be increased in the $INFA_HOME/tomcat/bin/infaservice.sh script under the JAVA_OPTS section.

                   

                  Regards,

                  Rajan