5 Replies Latest reply on Jul 31, 2020 3:46 AM by Prasenjit Saha

    Informatica PowerCenter service going down every 2-3 weeks

    Brinjit Velu Active Member

      Hello,

       

          We have an issue where in the Informatica server was going down several times because we are using only 1 CPU core each in Production and QA. I have attached logs for your review. From the attached logs can you identify the root cause and what should be done to prevent this? Please note that node_deda1x2488 is for QA and node_deda1x2489 is for Production.

       

      Regards,

       

      Brinjit

        • 1. Re: Informatica PowerCenter service going down every 2-3 weeks
          Asmita Kesarwani Seasoned Veteran

          Hi Brinjit,

           

          From the logs you shared we see the below error in both the servers:

           

          2020-07-26 01:00:47,282 FATAL [Domain Monitor] [DOM_10094] Cannot update the data for the master gateway node [node01_deda1x2489] within the refresh interval time [360000]. The node will not continue as a master gateway node. Verify that the connection to the domain configuration repository database is valid.

           

          Also, I see that in both the servers you have only one node.

           

          Also, I see that in both the servers you have only one node:

           

          1.High CPU usage on node:-

             If the thread responsible to update Domain database does not get the CPU

           

          2.High CPU usage on Database:-

              If database is not able to respond to the query in the timeout period due to CPU spike on Database Server or overload on Database

           

          3.Network issue

          If the query doesn’t reach to the Domain database within timeout period or the Domain does not receive the query response in the timeout period. This may happen if there is any Network issue at that timeframe.

           

          Since you mentioned you have only one CPU, it may be the reason of the issue, so please go ahead and increase the same.

           

          Also, I see that in both the servers you have only one node.

           

          As a workaround, masterDBrefreshinterval property can be set due to the network latency between database and server.

          The underlying issue must be resolved to avoid this in the future.

          You can set the MasterDBRefreshInterval custom parameter as follows:

          - Log in to the Administration Console

          - Click on the Domain on the Left-Hand navigation

          - On the Right-Hand Navigation click on Properties

          - Scroll to the bottom of the Properties page and edit the Custom Properties

          Under Properties , go to Custom Properties > Edit

          - Specify the parameter name MasterDBRefreshInterval and the value in seconds (30 or lesser is recommended).

          Example

           

          MasterDBRefreshInterval=15 (or higher)

          - Click OK.

          - Restart the domain.

           

          For more information, please refer the following the KB #145762 , #158429, #501330 articles.

           

          Kindly let me know if you have any queries on the same.

           

          Thanks,

          Asmita

          • 2. Re: Informatica PowerCenter service going down every 2-3 weeks
            Brinjit Velu Active Member

            Hi Asmita,

             

               I have adjusted MasterDBRefreshInterval = 30 as suggested and restarted the domain. Will monitor the system and keep you posted.

             

            Regards,

             

            Brinjit

             

             

             

             

            • 3. Re: Informatica PowerCenter service going down every 2-3 weeks
              Asmita Kesarwani Seasoned Veteran

              Hi Brinjit,

               

              Please update the same and monitor. Also, as I suggested there can be a possibility that there is a genuine requirement of the CPU. So, please check if you can increase the same as well.

               

              Thanks,

              Asmita

              • 4. Re: Informatica PowerCenter service going down every 2-3 weeks
                Asmita Kesarwani Seasoned Veteran

                Hi Brijit,

                 

                Additionally, once you increase the MasterDBRefreshInterval value, it will just make your System more resilient. So, in case if the issue occurs again after increasing this value we would recommend you to check on the CPU side as well.

                 

                Thanks,

                Asmita

                • 5. Re: Informatica PowerCenter service going down every 2-3 weeks
                  Prasenjit Saha Active Member

                  Hello Velu,

                   

                  FYI,

                  Pros and cons for increasing the MasterDbRefreshInterval is as follows:

                   

                  Pros:

                  It increases the resilience to communication issues with Domain DB.

                   

                  Cons:

                  It will delay the mastergateway node fail over to a different gateway node.

                   

                   

                   

                  Pros and cons for increasing the MasterUpdateTimeInterval is as follows:

                   

                  Pros:

                  It increases resilience to inter-node communication issues.

                   

                  Cons:

                  It will delay detection of a node fail over by mastergateway node.

                   

                  So, use should be aware of the above points while changing these properties.

                   

                  Hope this helps

                   

                  Thanks,

                  Prasenjit