11 Replies Latest reply on Jul 20, 2021 1:36 PM by Fabio Matos

    Custom Data Lineage Resource

    Valeria Dima New Member

      We have created the Data Lineage in EDC through the definition of a Custom Lineage Resource through the upload of the .csv template and we would need to display in the EDC Lineage and Impact tab, in addition to tables and columns read from Oracle, also some objects that allow us to map external element not coming from any other application.

       

      Is it possible to create a fictitious resource from scratch in EDC that allows us to do this?

        • 1. Re: Custom Data Lineage Resource
          Mayank Raoka Seasoned Veteran

          Hello Dima,

           

          yes you can create dummy resource as as custom resource to pull any external element in this case.Where your custom metadata will act as your external element in that resource.

           

          Please view this link for pulling custom metadata.

           

          How to bring custom metadata in EDC (ETL example)

           

          Custom Metadata Model in EDC

           

          Hope this helps.

           

          Regards,

          Mayank Raoka

          • 2. Re: Custom Data Lineage Resource
            Darren Wrigley Guru

            right - create a custom resource, either using a model you create, or re-use an existing model.

            you can also combine the custom lineage into that custom resource.

            • 3. Re: Custom Data Lineage Resource
              Bharat Rallapati Active Member

              Yes - You can re use an existing custom lineage template for custom resource

              • 4. Re: Custom Data Lineage Resource
                Valeria Dima New Member

                Thank you for your response.

                 

                What kind of model should I use to create a custom resource and link my custom lineage to it?

                • 5. Re: Custom Data Lineage Resource
                  Darren Wrigley Guru

                  that depends on what structure you need to use. 

                  you can either create a custom model, and include the classtypes you need to represent, or re-use an existing model (like the relational model used by most rdbms scanners)

                  • 6. Re: Custom Data Lineage Resource
                    Valeria Dima New Member

                    Another question:

                     

                    I have created a dummy resource by creating a file system resource scanner, with dummy parameter values, and executing the same, which caused the scanner to fail and register an empty resource in the Catalog UI.

                     

                    It is possible to link this empty resource into our custom data lineage created via through a Custom Data Lineage resource by uploading the .csv file?

                    • 7. Re: Custom Data Lineage Resource
                      Fabio Matos New Member

                      Hey guys,

                       

                      I have a related issue, we're trying to make a custom lineage using de API between an object Table and a Glue Table, but it simply doesn't work.

                       

                      Does anyone had the same issue?

                      • 8. Re: Custom Data Lineage Resource
                        Darren Wrigley Guru

                        @fabio - can you be more specific?

                        what is not working?  what is the error you are getting?

                        what is the relationship type you are trying to use?  (assume core.DataSetDataFlow)

                        what is the api endpoint you are using and the payload you are passing?

                        • 9. Re: Custom Data Lineage Resource
                          Fabio Matos New Member

                          Hi Darren,

                           

                          Thanks for your attention.

                           

                          I tried several relationship types, core.DataSetDataFlow was the first one I tried.

                           

                          I'm using API ../access/swagger-ui.html#/data-controller-v-1/updateObjectsUsingPATCH

                           

                          I tried this :

                          {

                          "providerId": "enrichment",

                          "modifiedBy": "administrator",

                          "updates": [

                          { "newSourceLinks":

                          [ { "objectId": "DataLake://Hive Metastore/ref_mongo/customers",

                          "properties":

                          [

                          { "attrUuid": "core.targetAttribute",

                          "value": "core.DataSetDataFlow"

                          }

                          ],

                          "associationId": "core.DataSetDataFlow"

                          }

                          ],

                          "deleteFacts": [ ],

                          "newFacts": [ ],

                          "deleteSourceLinks": [ ],

                          "id": "GLUE_EXP://exp/customers"

                          }

                          ]

                          }

                           

                          It runs ok, i got a 200 response, but the lineage does not appear between the objects. The source object is type Table and destination is GlueTable.

                           

                          This other command, for example, works fine and create the lineage perfectly, both source and destination are Table objects:

                          {

                          "providerId": "enrichment",

                          "modifiedBy": "administrator",

                          "updates": [

                          { "newSourceLinks":

                          [ { "objectId": "DataLake://Hive Metastore/ref_mongo/products",

                          "properties":

                          [

                          { "attrUuid": "core.targetAttribute",

                          "value": "core.DataSetDataFlow"

                          }

                          ],

                          "associationId": "core.DataSetDataFlow"

                          }

                          ],

                          "deleteFacts": [ ],

                          "newFacts": [ ],

                          "deleteSourceLinks": [ ],

                          "id": "DataLake://Hive Metastore/exp/products"

                          }

                          ]

                          }

                          • 10. Re: Custom Data Lineage Resource
                            Darren Wrigley Guru

                            I just tried the same thing - with a minimal entry for the payload:-  (snowflake and google bigquery tables - the scanner type should not matter)

                             

                            {

                              "providerId": "enrichment",

                              "updates": [

                                {

                                  "id": "snowflake_all_standard://TEST_DB/PUBLIC/CRM_CUSTOMER_MAIN",

                                  "newSourceLinks": [

                                    {

                                      "associationId": "core.DataSetDataFlow",

                                      "objectId": "biqquery-datacatalog-sandbox://edc-datacatalog-sandbox/tpcds/date_dim"

                                    }

                                  ]

                                }

                              ]

                            }

                            it worked ok and the lineage shows the relationship.  i could also have used newDestinationLinks