1 2 Previous Next 15 Replies Latest reply on Sep 21, 2021 6:34 PM by Thiru S

    EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.

    Anish Gupte Active Member

      Hi Team,

       

      I am getting an error while using the export utility to back up the catalog contents in our 10.4.1.3 environment.  As per the upgrade guide I copied the application.properties file from the following location under the 10.4 INFA_HOME directory : -

       

      /INFA_HOME/logs/<node_name>services/CatalogService/<CatalogServiceName>/spark

       

      to the following location :

       

      /<extracted 10.5.x installer files>/properties/util/upgrade/EDC/

       

      I executed the java command to generate the export file  : java -jar export.jar export

       

      I am seeing the following error : --

       

      ERROR   2021-09-13 15:40:26,197 612345  com.infa.products.edc.xdoc_exporter.PhoenixXdocExporter [main]  Error creating bean with name 'changeNotificationExporter' defined in URL [jar:file:/infa_edc_shared/10.5.0/install/properties/utils/upgrade/EDC/export.jar!/com/infa/products/edc/xdoc_exporter/change_notif/ChangeNotificationExporter.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'modelExt' defined in com.infa.products.edc.xdoc_exporter.SpringConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.infa.products.ldm.data.utils.ModelExt]: Factory method 'modelExt' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'modelQueryClient' defined in com.infa.products.edc.xdoc_exporter.SpringConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.infa.products.ldm.ingestion.modelquery.ModelQueryClient]: Factory method 'modelQueryClient' threw exception; nested exception is java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=40, exceptions:

      Mon Sep 13 15:30:14 CDT 2021, RpcRetryingCaller{globalStartTime=1631565014667, pause=100, retries=40}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't get master address from ZooKeeper; znode data == null

      Mon Sep 13 15:30:14 CDT 2021, RpcRetryingCaller{globalStartTime=1631565014667, pause=100, retries=40}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't get master address from ZooKeeper; znode data == null

      Mon Sep 13 15:30:14 CDT 2021, RpcRetryingCaller{globalStartTime=1631565014667, pause=100, retries=40}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't get master address from ZooKeeper; znode data == null

       

      The knowledge base unfortunately does not have any entry for this issue specifically occurring during the catalog contents back up process using the export utility, but i did find an article for fixing the  'Can't get master address from Zookeeper; znode data = = null' issue. - not sure if this is an accurate fix in this scenario.

       

      Support

       

       

      On a related note, we also took a back up using the LDM contents command on the CLI, and we were able to create a zip file , but it was barely a little under 2 MB which to me indicates that somehow there are no cluster contents at all in this environment.  Could this be a reason why the export utility is also failing ? If anyone has seen this before, please let me know your thoughts!

       

      Thanks,

      Anish.

       

       

       

       

       

        • 1. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
          Thiru S Active Member

           

          Please verify whether the Catalog Service was in running state, when the 'export' utility was run. Observed error would be seen, when the Catalog service is not in RUNNING state (or) if some of the Catalog service related YARN applications like 'hbase'/'solr' are not running in the Hadoop cluster.

           

          For successful export, 'Catalog service' should be in 'Running' state, and it should be put in 'READONLYMODE' . If the Catalog service is running fine, then we should be able to see the applications - hbase, solr - in the YARN Resource manager UI of Hadoop cluster.

           

          To put the Catalog service in READONLYMODE, refer to the following KB article:

           

          https://knowledge.informatica.com/s/article/HOW-TO-Enable-the-read-only-mode-for-the-Catalog-Service?language=en_US

           

           

           

          • 2. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
            Anish Gupte Active Member

            Hi Thiru/ Team,

             

            Thank you for your response. So, here is what we did : -

             

            1. We tried to use an existing LDM contents back up file (of the same version 10.4.1 from a different env) and restore contents using the infacmd.sh LDM RestoreContents from  the CLI.

             

            2. The restore contents command failed with an exception that contents already exist in the HDFS directory for the cluster service.

             

            ICMD_10033] Command [restorecontents] failed with error [[LDMCMD_20030] Contents already exists either in the following hdfs directory [/Informatica/LDM/ICS_EDC_PROD] or in the zookeeper node [/Informatica/LDM/] on the target cluster. Hence stopping service restore.

             

            Thus, we ran a cleanCluster command and restarted the Informatica Domain.

             

            At this point, our aim is to restore contents into the IHS using the existing back up file (from step 1). then create a back up of the LDM contents using the export utility, and then proceed with the in place upgrade.

             

             

            However, we are now having issues starting up the IHS with the following error : -

             

            2021-09-14 01:49:38,670 [localhost-startStop-1] ERROR com.infa.products.ihs.beans.application.ClusterListener- com.infa.products.ihs.service.exception.InfaHadoopServiceException: [InfaHadoopServiceException_00001] Ambari request to Start of the 'HDFS' is aborted or failed. View the log files of Ambari server and Ambari client for more information

            1. com.infa.products.ihs.service.exception.InfaHadoopServiceException: [InfaHadoopServiceException_00001] Ambari request to Start of the 'HDFS' is aborted or failed. View the log files of Ambari server and Ambari client for more information

            at com.infa.products.ihs.service.ambari.AmbariInfaHadoopClusterService.pollAmbariRequests(AmbariInfaHadoopClusterService.java:1718)

            at com.infa.products.ihs.service.ambari.AmbariInfaHadoopClusterService.submitStartServiceReqPoll(AmbariInfaHadoopClusterService.java:1456)

            at com.infa.products.ihs.service.ambari.AmbariInfaHadoopClusterService.startServices(AmbariInfaHadoopClusterService.java:1360)

            at com.infa.products.ihs.service.ambari.AmbariInfaHadoopClusterService.startCluster(AmbariInfaHadoopClusterService.java:1250)

            at com.infa.products.ihs.service.ambari.AmbariInfaHadoopClusterService.createAndStartCluster(AmbariInfaHadoopClusterService.java:348)

            at com.infa.products.ihs.service.ambari.AmbariInfaHadoopClusterService.enableCluster(AmbariInfaHadoopClusterService.java:321)

            at com.infa.products.ihs.beans.application.ClusterLauncher.lauchCluster(ClusterLauncher.java:22)

            at com.infa.products.ihs.beans.application.InfaHadoopApplication.launchCluster(InfaHadoopApplication.java:650)

            at com.infa.products.ihs.beans.application.ClusterListener.onApplicationEvent(ClusterListener.java:40)

            at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172).

             

             

            Any thoughts on this issue ? My thought is maybe we need to run the restore contents command before starting up the IHS?

             

            Any help is greatly appreciated.

            • 3. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
              Thiru S Active Member

              Hi Anish,

               

                 As the cleanup has been performed, during the IHS startup, a new cluster would be setup.

               

              'EDC Restore contents' command should be run, after the IHS has been started successfully, but before starting the 'Catalog service'.

               

              From the currently seen error, it appears like during the new cluster setup, HDFS is service is having some issues. To better understand and debug the same with HDFS service, we would need to review the HDFS service logs.

               

              In the current scenario, I recommend to create a new technical case with Informatica GCS team, by including following details, so as to analyze and fix the HDFS related issue:

               

              • IHS.log from '$INFA_HOME/logs/[node]/services/InfaHadoopService/[Service_Name]' location from Informatica Domain machine.
              • Zip and share the logs corresponding to HDFS service from Hadoop cluster machine.

              Default Log Location: '/var/log/hadoop/hdfs' location.

               

              In case if IHS is using custom Hadoop install directory through custom property - 'IcsCustomOptions.ihs.hadoop.dir', then HDFS logs can be downloaded from '$IHS_HOME_DIR/var/log/hadoop/hdfs' 

               

               

              E.g.

               

              If  '/app/informatica/hadoop ' is used as custom install location for IHS, then HDFS logs should be collected from '/app/informatica/hadoop/var/log/hadoop ' from Hadoop cluster machine

              • 4. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                Anish Gupte Active Member

                Thanks Thiru - we have opened a case and are working with Support on this issue . I did have a question regarding the 10.4 to 10.5 upgrade process, specifically using the Export Utility.

                 

                The standard approach as per the upgrade documentation involves creating a back up of the catalog contents in the 10.4 environment using the export utility (prior to the upgrade), and  then restoring from this back up file post upgrade to 10.5 into the upgraded catalog using the infacmd.sh migrateContents command on the server

                 

                However, in our situation, at this time we have executed the cleanCluster operation in our 10.4 env and hence we not have any contents in our HDFS cluster(s). As of now, we are in  the process of bringing up our Hadoop Service,, after which we will restore contents into the cluster from a 10.4 back up from a different environment and then proceed with the upgrade.

                 

                But, what if we use the following approach : -

                 

                1. In our 10.4 env ( Lets call this Env A), once the IHS comes up, instead of restoring LDM contents we will proceed directly with the in place upgrade to 10.5.

                2. Post Upgrade, there will be no catalog contents and the Catalog Service will be in read only mode.

                3. At this point, we use the export utility in a different 10.4 env (Lets call this Env B, hosted on a different domain) to create a back up file.

                4. We then move this file (created in Env B) to Env A and run the infacmd.sh migrateContents command. Thus essentially we will be migrating LDM contents from a back up file created in Env B into our upgraded catalog in Env A.

                 

                Since we be migrating contents across domains, will this create any issues? Is this approach feasible or even advisable ?

                 

                Let me know your thoughts.

                 

                Thanks,

                Anish.

                • 5. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                  Thiru S Active Member

                  Hi Anish,

                  Yes, the mentioned approach of migrating the Catalog contents using 'Export' utility from one Environment (say, Env B) to another (say, Env A) would be possible.

                   

                  However, for such use case, before running 'infacmd ldm migrateContents', we have to perform backup, restore & upgrade of MRS, associated with EDC as well.

                   

                   

                  i.e., We have to take 10.4.x backup of EDC MRS from source environment (say, Env B), restore the MRS backup into target environment (say, Env A), upgrade the MRS contents and then have to proceed with 'migrateContents' command execution.

                   

                   

                  'migrateContents' command execution would be interlinked to EDC resources, whose details are stored in the MRS. In case, if there is mismatch in the resources between the '10.4.x catalog export' output & actual resources details in the EDC MRS, then execution would fail. To avoid such issues, we have to do backup & restore of EDC MRS as well.

                   

                  If you proceed with that approach of running 'export' utility in different EDC environment, then we don't need to again bring up IHS in the environment, to be upgraded to 10.5.x version. As in 10.5.x, we won't need IHS, you can run 'infacmd ihs cleancluster' to clean up the Hadoop related packages and then can proceed directly with the in-place upgrade.

                  • 6. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                    Anish Gupte Active Member

                    Hi Thiru ,

                     

                    Thanks - that definitely makes sense. So then, consider the following scenario : -

                     

                    1. We create a .zip back up file using the infacmd.sh LDM BackupContents command in Env B.

                    2. We restore this back up using the infacmd.sh restoreContents command in Env A. (now the cluster has in Env A has contents from back up file created in Env B)

                    3. Then we create an export of the LDM contents in Env A

                    4. Perform an in-place upgrade in Env A.

                    5. Post upgrade , we migrate contents using the export created in step 3.

                     

                     

                    We are not touching the MRS at all in this case. Will this approach work ? This is what we are aiming to do.

                     

                    Thankls,

                    Ansih.

                    • 7. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                      Thiru S Active Member

                      Hi Anish,

                       

                        Even when we move EDC contents between environments using 'ldm backupContents/restoreContents' command, we would need the associated MRS backup to be restored in target environment. Otherwise, after restore, we will not able to search the assets in EDC Catalog UI. When we upgrade such environment setup, where only EDC content is moved, without its dependent MRS content, it could cause different side effects.

                       

                      Thumb rule is whenever we move EDC contents between environments (either using 'ldm backupContents' or with 'ldm migrateContents' option ), we need to ensure that the associated MRS contents are also moved to target environment, for EDC to work as expected.

                       

                       

                      • 8. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                        Anish Gupte Active Member

                        Thanks Thiru - We have taken latest backups of both LDM and MRs and are proceeding with the content restore process.

                         

                        A quick question - I kicked off the LDM restore operation but my putty session timed out after a few minutes - I have changed my keep alive settings now, but do I need to execute the restore command again , or is the process running ? If so, where can I see the log file for the LDM restore ( assuming it is still running ).

                         

                        Thanks for all your help

                        • 9. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                          Thiru S Active Member

                          Hi Anish,

                           

                          By default, all infacmd commands including 'backupcontents/restorecontents'  execution, would be started as a foreground process. Hence if the session gets timed out, then the process would also get stopped/killed. Also, the commands output would be logged by default only stdout console and there would be no log files generated.

                           

                          To confirm if there are any active process, you can run 'ps' command as below:

                           

                          ps -ef | grep -i restorecontents | grep -v grep

                           

                          You can start the backup/restore in background using 'nohup' & also capture the output to a file, using commands similar to below:

                           

                          Backup

                           

                          nohup ${infa_domain_home}/isp/bin/infacmd.sh ldm BackupContents -dn ${infa_domain_name} -un "${infacmd_user_name}" -sdn "${infacmd_user_security_domain}" -sn "${infa_service_name}" -hsd "${edc_hdfs_full_path}" -of "${current_domain_ldm_backup_fullname}" > /tmp/nohup_edc_backup_contents.out 2>&1 &

                           

                          Restore

                           

                          nohup ${infa_domain_home}/isp/bin/infacmd.sh ldm RestoreContents -dn ${infa_domain_name} -un "${infacmd_user_name}" -sdn "${infacmd_user_security_domain}" -sn "${infa_service_name}" -if "${current_domain_ldm_backup_fullname}" > /tmp/nohup_edc_restore_contents.out 2>&1 &

                           

                           

                          While using commands as mentioned above, output would be captured in the mentioned files.

                           

                           

                           

                           

                          • 10. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                            Anish Gupte Active Member

                            Yes Thiru - I reran the LDM restoreContents command using the nohup command, and the restore completed successfully in about 3.5 hrs. Thank you for your input.

                             

                            We are now running the re-index contents operation from the Admin Console --> Catalog Service Actions -->Re-index contents.

                             

                             

                            The reindexing is running now for the last 3 hours, and  i can see that the logs are moving forward, but the reindexing progress is still at 0%.

                             

                            We have nearly 13 million assets, so we do expect it to run a long time, but since it has not moved past 0%, is this something to be concerned?

                             

                            Here is a portion of the log from the server :-

                             

                            2021-09-16 15:15:59,205 [ReindexProgress] INFO  - [0%] reindexing completed

                            2021-09-16 15:16:09,209 [ReindexProgress] INFO  - [0%] reindexing completed

                            2021-09-16 15:16:19,212 [ReindexProgress] INFO  - [0%] reindexing completed

                            2021-09-16 15:16:29,216 [ReindexProgress] INFO  - [0%] reindexing completed

                            2021-09-16 15:16:39,219 [ReindexProgress] INFO  - [0%] reindexing completed

                            2021-09-16 15:16:49,222 [ReindexProgress] INFO  - [0%] reindexing completed

                            2021-09-16 15:16:59,226 [ReindexProgress] INFO  - [0%] reindexing completed

                            2021-09-16 15:17:09,229 [ReindexProgress] INFO  - [0%] reindexing completed

                            2021-09-16 15:17:19,233 [ReindexProgress] INFO  - [0%] reindexing completed

                            2021-09-16 15:17:29,236 [ReindexProgress] INFO  - [0%] reindexing completed

                             

                             

                            Thanks,

                            Anish.

                            • 11. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                              Thiru S Active Member

                              Hi Anish,

                              Yes, the re-indexing operation would take time depending on volume of assets in the catalog. In certain cases, the progress percentage might be shown as 'zero', though it might be actively going on.

                               

                              You can continue to monitor for some more hours. In case if the operation is not finished even later, you can raise case with Informatica GCS to debug the same.

                              • 12. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                                Chris Beyer New Member

                                Hi Thiru, Anish's colleague here -

                                 

                                We understand that any LDM backup must coincide with an MRS backup in order to ensure everything is in sync. Theoretically, this would mean that a full backup could be done from a 10.4 "Env B" (in Anish's example) to a 10.5 "Env A" if both the MRS and LDM could be backed up and restored across versions. Clearly there is a way to do this for the LDM, but our impression is that the MRS cannot be backed up and restored across versions. This would mean that it is impossible to backup and restore 10.5 "Env A" from 10.4 "Env B".

                                 

                                Can you confirm whether this is true?

                                • 13. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                                  Thiru S Active Member

                                  Hi Chris,

                                  MRS Backup can be restored across different versions directly (say, 10.4.0 -> 10.5.x) . However, due to architectural changes related to Catalog service (i.e. moving away from Hadoop platform cluster to non-Hadoop platform cluster), LDM backup cannot be restored directly from 10.4.x to 10.5.x version

                                  • 14. Re: EDC 10.4.1 to 10.5.0.1.1 in place upgrade - Export Utility issue.
                                    Anish Gupte Active Member

                                    Hi Thiru,

                                     

                                    Thank you for your response. I wanted to clarify your post : -

                                     

                                    1. When you refer to the LDM backup -  I am assuming you are referring to the LDM backup which is generated using the infacmd.sh LDM backupContents command on the server - which is created as a.zip file - this LDM backup cannot be restored directly from 10.4.x to 10.5.x?

                                     

                                    However, as per my understanding, the LDM backup which is generated using the export utility located at the following location : -

                                    <extracted 10.5 installer files>/properties/util/upgrade/EDC/ , is the backup which is used to migrate the catalog contents from 10.4.1 to 10.5.x , meaning -

                                    - We create a backup for the 10.4.1 LDM contents using this utility.

                                    - Post-upgrade, we use this back up and execute the infacmd.sh migrateContents command from $INFA_HOME/isp/bin location (on the 10.5 gateway node) to migrate 10.4.1 contents from the Hadoop platform cluster to the MongoDB platform cluster.

                                     

                                    Please let me know if this is accurate.

                                     

                                     

                                    2.  Also, to your point about the MRS Backup and restore process, if I have an MRS A which is configured in my 10.4.x environment, can I create a backup for this from the Admin Console and restore that backup directly into MRS B which is configured in my 10.5.x environment ?

                                     

                                    When I used to perform upgrades for PC and DQ, I did this 1 of 2 ways (lets assume we are talking about a 10.4.1 to 10.5.x upgrade) -  either you do an in-place upgrade- where you upgrade the domain through the installer and then upgrade the service contents from the admin console on startup, or , in case of a parallel upgrade approach , you create a new MRS in the 10.4 domain with 10.4 contents pointing to the 10.5 database , and then create the same MRS in the newly installed 10.5 domain - then on startup, the  system will prompt for an upgrade of the service contents. I have never taken backup of 10.4 MRS contents  and restored directly into a 10.5 MRS instance.

                                     

                                    I am quite certain that Informatica does not allow direct MRS backup and restore between different versions. Please do let us know to the best of your knowledge if this is no longer the case and/or if Informatica has now updated its code base to allow such a restore.

                                     

                                    As always, we really appreciate your help.

                                     

                                    Thanks,

                                     

                                    Anish.

                                    1 2 Previous Next