1 Reply Latest reply on Apr 23, 2021 6:16 AM by Shailesh Khuperkar

    Hive Resource pulling Reference Data Sets

    Suvrat Rai New Member

      Hi All,

       

      I have two questions around EDC.

      We have created a Cloudera Hive resource on Enterprise Data Catalog v10.4.

       

      1. We have selectively added multiple schemas and added table names under Metadata Load Settings tab on the Schema & Source Metadata Filter options. After resource execution, we are seeing few reference datasets being created on the system along with the regular tables. Can anyone explain, why and under what conditions these reference data sets are getting created while scanning the Hive resource?
      2. Also, we have a scenario where one table is common across multiple schemas. I want to pull the table metadata from one schema and not the other. We have already tried putting in SchemaName.TableName on the 'Source Metadata Filter' option, but that does not seem to be working and we are not able to pull any datasets in this way. Hence we had to put in table names under 'Source Metadata Filter' option and select the schema separately. Is there any way we can achieve this type of filtering?

       

      Thanks & Regards,

      Suvrat

        • 1. Re: Hive Resource pulling Reference Data Sets
          Shailesh Khuperkar Seasoned Veteran

          Hi Suvrat,

                      Regarding your 1st question, EDC creates reference resources if there are references from your hive assets (tables/column/) or resource to other assets.

          Extracting reference assets in the catalog provides the following benefits:

          • Search for reference resources and assets in Enterprise Data Catalog.

          • Filter the search results based on reference resource and assets.

          • View lineage for a primary resource that includes reference resources and assets.

          • View Lineage, Impact, and Control Summary for reference assets.

          • Filter lineage based on reference assets.

          • Annotate and enrich reference assets with custom attributes and business terms.

          • Perform a bulk export or import of the custom attributes, business term associations, and other enrichments on reference assets.

          • View custom lineage between reference resources.

           

          You can open any one of the reference assets and check its lineage and impact so that it will show its parent asset for which reference resource is created.

           

          Regarding 2nd query, I don't think there is any way to achieve your requirement. I would suggest to raise support case to get more assistance on the same.

           

          Thanks,

          Shailesh

          1 of 1 people found this helpful