1 Reply Latest reply on Nov 17, 2021 5:51 AM by Srini Pai

    EDC Catalog Services and DIS

    John Quillinan Guru

      Persistent connections are links that do not close when the execution of your script ends. When a persistent connection is requested, a programs checks if there's already an identical persistent connection (that remained open from earlier) - and if it exists, it uses it.  When I run the Catalog Services to extract source metadata, does EDC maintain persistent connection with the database until Metadata Load completes?  Or does the connection close immediately after results are returned?

       

      In the case of Data Integration Services (DIS), when profiling is performed. If a table contains more number of columns, not all columns will be read using a single query.   Based on the DIS properties (like max concurrent columns, max memory per request, concurrent profiling etc.,) multiple mappings will be triggered to process data from different set of columns, which eventually issues multiple queries to retrieve all the columns from the tables.

      Example: If a table contains 25 columns, 5 mappings will get triggered and 5 select queries will be issued to the DB to select all 25 columns data.

       

      If multiple mappings are triggered, does DIS open one connection to a database?  Does DIS open multiple connections to the database, and close each connection as result sets are returned?

        • 1. Re: EDC Catalog Services and DIS
          Srini Pai New Member

          Hello John

           

          During the metadata load task we see several stages like test connection, metadata extraction , staging and publish, These are self explanatory.

          Connections to the database is established and disconnected during the test connection and metadata extraction task only. The number of connections created is determined by the driver or client behavior(JDBC/ODBC or Native clients) , From the applications multiple queries are fired to the database to run in parallel and there could be one or more connections established 

           

          During profiling the behavior followed is similar, Every table would be assigned a separate profile task and each profile tasks would have multiple mapping tasks, the number of mapping tasks would be decided runtime by DIS and is dependent on number of columns/rows and datatypes.

          From DIS you can tweak the properties to decide how much parallel mappings or connections could be created to the database. (This might end up with performance issues or hogging up resources in the database end so should be done judiciously)

           

          If multiple mappings are triggered, does DIS open one connection to a database? 

          -- NO. This would be one or more connection per mapping. 

          Does DIS open multiple connections to the database, and close each connection as result sets are returned?

          -- Yes, Once the Read task or Reader Thread in the mapping is complete the connection to the database would be given away.

           

           

          Regards

          Srinivas

          1 of 1 people found this helpful