9 Replies Latest reply on Sep 15, 2020 4:39 PM by Shailesh Khuperkar

    EDC - Hive View

    Alice Lim New Member

      Hi, trying to understand if there is any limitation in bringing in Hive view? Manage to see the asset under the Resource but wasn't able to see the columns and lineages.

       

      Thank you.

        • 1. Re: EDC - Hive View
          Likhitha Jami New Member

          Hi Alice,

           

          Please answer below questions:

          What is the EDC version?

          What is the Hive version along with cluster distribution?

          Where you able to see the hive-view columns and lineage in the Catalog UI earlier?

           

          Thanks,

          Likhitha Jami

          • 2. Re: EDC - Hive View
            Shailesh Khuperkar Seasoned Veteran

            Hi Alice,

                      There are no limitations as such on bringing hive view columns and lineages in EDC. Many times EDC don't bring view lineage and columns if it is not able to parse view statements. Such parse error can be found in scanner logs. Is this happening for any particular view or all views?

             

            Thanks,

            Shailesh

            • 3. Re: EDC - Hive View
              Alice Lim New Member

              What is the EDC version?

              >> 10.4

               

              What is the Hive version along with cluster distribution?

              >>HDP 3.1, Hive 3.1

               

              Where you able to see the hive-view columns and lineage in the Catalog UI earlier?

              >> I only to see hive view columns and lineages where it is been used for propagating the target table. For these views that are being used for propagating the target table, additional Reference Data Set were also created.

               

              For example:

              I have 4 views (vw_a/b/c/d) and selected columns from the 2 of the views (vw_a/b) were used to ingest into a target Hive table. In EDC, I'm able to see 4 views + 2 reference data set. Only selected columns in vw_a and vw_b that were used to ingest into the target Hive table were shown on EDC.

              • 4. Re: EDC - Hive View
                Alice Lim New Member

                To be correct, I'm able to see all the views (as asset) under the resource. But, I only able to see hive view columns and lineages where it is been used for propagating the target table. For these views that are being used for propagating the target table, additional Reference Data Set were also created.

                 

                For example:

                I have 4 views (vw_a/b/c/d) and selected columns from the 2 of the views (vw_a/b) were used to ingest into a target Hive table. In EDC, I'm able to see 4 views + 2 reference data set. Only selected columns in vw_a and vw_b that were used to ingest into the target Hive table were shown on EDC.

                • 5. Re: EDC - Hive View
                  Shailesh Khuperkar Seasoned Veteran

                  Hi Alice,

                             Thanks for your information. From your above information, I understand you are able to see the lineage for data flow propagation through the views to other sources but you are not able to see the lineage for data flow from base tables to view. If my understanding is correct,

                  1. can you please check if there are any errors (parse/syntax errors) in scanner logs for those views?

                  2. Are base tables used for views part of same resource?

                   

                  If my understanding is not correct, screen shot from catalog UI would be helpful for better understanding.

                   

                  Thanks,

                  Shailesh

                  • 6. Re: EDC - Hive View
                    Darren Wrigley Guru

                    are the tables and views in the same schema (in hive i think that is a database), and were both of these schemas/databases included in the scan?

                    • 7. Re: EDC - Hive View
                      Alice Lim New Member

                      Hi Shailesh,

                       

                      Yes, the base table and view are harvested under the same resource.

                       

                      I reviewed the LDM.log and following error msg:

                       

                      2020-09-14 14:39:00,480 - INFO [DTF-ThreadGroup-6-thread-2:HeartbeatChecker$HeartbeatSender@304] - [DTF_0001] An error occurred in the underlying transport layer: [java.util.concurrent.ExecutionException: org.beepcore.beep.core.BEEPException: Unexpected SSL wrap result status: BUFFER_OVERFLOWjava.io.IOException: Unexpected SSL wrap result status: BUFFER_OVERFLOW]

                      1. com.informatica.pcsf.datatransport.DataTransportException: [DTF_0001] An error occurred in the underlying transport layer: [java.util.concurrent.ExecutionException: org.beepcore.beep.core.BEEPException: Unexpected SSL wrap result status: BUFFER_OVERFLOWjava.io.IOException: Unexpected SSL wrap result status: BUFFER_OVERFLOW]

                      at com.informatica.pcsf.datatransport.impl.DataTransportChannelImpl.sendRequest(DataTransportChannelImpl.java:92)

                      at com.informatica.pcsf.servicesframework.client.impl.HeartbeatChecker$HeartbeatSender.sendHeartbeat(HeartbeatChecker.java:290)

                      at com.informatica.pcsf.servicesframework.client.impl.HeartbeatChecker$HeartbeatSender.sendHeartbeatToRecipient(HeartbeatChecker.java:261)

                      at com.informatica.pcsf.servicesframework.client.impl.HeartbeatChecker$HeartbeatSender.run(HeartbeatChecker.java:229)

                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

                      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

                      at java.lang.Thread.run(Thread.java:748)

                      Caused by: org.beepcore.beep.core.BEEPException: org.beepcore.beep.core.BEEPException: Unexpected SSL wrap result status: BUFFER_OVERFLOWjava.io.IOException: Unexpected SSL wrap result status: BUFFER_OVERFLOWjava.util.concurrent.ExecutionException: org.beepcore.beep.core.BEEPException: Unexpected SSL wrap result status: BUFFER_OVERFLOWjava.io.IOException: Unexpected SSL wrap result status: BUFFER_OVERFLOW

                      at org.beepcore.beep.core.ChannelImpl.sendMSG(ChannelImpl.java:493)

                      at com.informatica.pcsf.datatransport.impl.DataTransportChannelImpl.sendRequest(DataTransportChannelImpl.java:88)

                      ... 8 more

                      Caused by: java.util.concurrent.ExecutionException: org.beepcore.beep.core.BEEPException: Unexpected SSL wrap result status: BUFFER_OVERFLOWjava.io.IOException: Unexpected SSL wrap result status: BUFFER_OVERFLOW

                      at java.util.concurrent.FutureTask.report(FutureTask.java:122)

                      at java.util.concurrent.FutureTask.get(FutureTask.java:206)

                      at org.beepcore.beep.core.ChannelImpl.sendMSG(ChannelImpl.java:473)

                      ... 9 more

                      Caused by: org.beepcore.beep.core.BEEPException: Unexpected SSL wrap result status: BUFFER_OVERFLOWjava.io.IOException: Unexpected SSL wrap result status: BUFFER_OVERFLOW

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession.sendFrame(TCPNonBlockingSession.java:467)

                      at org.beepcore.beep.core.ChannelImpl.sendFrames(ChannelImpl.java:1008)

                      at org.beepcore.beep.core.ChannelImpl.sendQueuedMessages(ChannelImpl.java:916)

                      at org.beepcore.beep.core.ChannelImpl.sendToPeer(ChannelImpl.java:881)

                      at org.beepcore.beep.core.ChannelImpl.sendMsgInternal(ChannelImpl.java:422)

                      at org.beepcore.beep.core.ChannelImpl.access$0(ChannelImpl.java:388)

                      at org.beepcore.beep.core.ChannelImpl$AsyncMessageSender.call(ChannelImpl.java:1351)

                      at org.beepcore.beep.core.ChannelImpl$AsyncMessageSender.call(ChannelImpl.java:1)

                      ... 4 more

                      Caused by: java.io.IOException: Unexpected SSL wrap result status: BUFFER_OVERFLOW

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession.ensureFullWriteAfterWrapping(TCPNonBlockingSession.java:892)

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession.ensureFullWrite(TCPNonBlockingSession.java:514)

                      at org.beepcore.beep.transport.tcp.TCPNonBlockingSession.sendFrame(TCPNonBlockingSession.java:459)

                      • 8. Re: EDC - Hive View
                        Alice Lim New Member

                        Hi Darren,

                         

                        Both tables and views are created in the same db and  harvested into EDC under the same resource.

                        • 9. Re: EDC - Hive View
                          Shailesh Khuperkar Seasoned Veteran

                          Hi Alice,

                                     Thanks for sharing the LDM.log error. This error from LDM.log is common error and not related to hive resource and can be ignored. We should be able to see the Hive scanner logs using below steps.

                           

                          1. Open your resource in LDM Admin

                          2. go to Monitoring tab

                          3. Click on 'Log Location' in front of metadata load step

                          4. On log page, search for 'stdout'

                          5. under 'stdout' logs, click on 'Click here for full logs' to get full logs and send us the same full logs.

                           

                          Once we get the scanner logs using above steps, we can search for view name to see if there are any errors about the views in the scanner logs.

                           

                          I hope this helps you.

                           

                          Thanks,

                          Shailesh