5 Replies Latest reply on Mar 3, 2020 2:07 PM by user101600

    Informatica Production env is down

    Kittu Babu Guru

      Our Informatica Environment is down. When I verified the node.log, I can able to see the below info.

       

      Please note that our Informatica is running on a single node domain.

       

      2019-09-09 03:09:49,633 ERROR [Monitor Guaranteed Message File Thread] [LGS_10048] The following error occurred while logging to Log Service: [[DOM_10022] The master gateway node for the domain is not available. Electing another master gateway. Wait for the election of the master gateway node to  complete. If the problem persists, verify that the master gateway node is running.].\n

      com.informatica.isp.domainservice.exceptions.ISPMasterNotAvailableException: [DOM_10022] The master gateway node for the domain is not available. Electing another master gateway. Wait for the election of the master gateway node to  complete. If the problem persists, verify that the master gateway node is running.

       

      What could be the issue from the above message ? Could you pls help.

        • 1. Re: Informatica Production env is down
          Nico Heinze Guru

          This message will appear every time when the current master gateway (which in your case is your only gateway node) can't reach the domain repository DB in time. The reason is that all gateway nodes periodically check a certain table in the domain repository for a recent update; the current master gateway updates this particular table every few seconds, and if it cannot do so, then this DB entry becomes "too" old, and all gateway nodes then try to negotiate with one another which one will become the next master gateway.

          This is a standard safety measure built into the Informatica domain to make sure that there are no long delays after a gateway node ceased to work before everything will be up and running again. And so this is done even in single-node installations.

           

          Probably there was some significant delay in network access to the database server hosting the domain repository. Try to find out - using the other log files of the Tomcat / Apache installation of your Informatica domain, DB logs, and network logs - where in the network this delay occurred, why it occurred, and how to remedy this situation.

           

          As of my knowledge this interval for updating this particular DB entry is hard-coded in the Informatica installation (8 seconds, if I recall correctly) and cannot be changed in any way. You have to make sure that there are no delays between the Informatica node and the domain repository DB server longer than these eight seconds.

           

          Regards,

          Nico

          • 2. Re: Informatica Production env is down
            Anukrati Tiwari New Member

            Hi Kittu,

             

            There could be a lot of reasons for this error message. You'd see a similar message when you try to start up the services too.

            To find the reason why this failure occurred, look for the time when domain deinit was called in catalina.out and check for the corresponding error message in the node.log( Do check for fatal error messages in the node.log)

             

            Thanks,

            Anukrati

            • 3. Re: Informatica Production env is down
              Neelanjan Ghosh Active Member

              Hello Kittu Babu,

              You can look for any FATAL messages in your INFA_HOME/logs/<node name>/node.log and share it here for further analysis.

               

               

              Thanks,

              Neelanjan

              • 4. Re: Informatica Production env is down
                Pavan kumar Guru

                Hi Kittu Babu,

                 

                If you need help from us to review the logs, please upload node.log and catalina.out here so that we can check and let you know what could have caused the node down.

                 

                Thank you,

                Pavan

                • 5. Re: Informatica Production env is down
                  user101600 Seasoned Veteran

                  for starters if this is truly production and you need an immediate response, Then i would highly recommend opening a new case with Informatica support.

                   

                  Generally speaking this is a very generic error message that can mean a lot of different things.

                  Its the other errors i n the logs that are more relevant.

                   

                  But going off the only piece of infromation that is available, most of the time this type of scenario would occur due to any of the following reasons:

                  1- There was a network issue between the node and domain repo DB.

                  2- The DB hosting the domain repo DB was not available.

                  3- The DB hosting the domain repo DB was to busy and unable to respond to the update query within the allocated timeout period.

                  4- The box where the PC node runs on was experiencing some sluggish performance.

                   

                  From a PC perspective the only thing that you can do is try increase the PC related timeouts in order to be more tolerant of any system resources issues, or network related issues that PC might encounter when communicating to the domain repo.