3 Replies Latest reply on Oct 4, 2021 12:31 PM by Srinivas Pai

    JDBC row limit issue

    Brian T Vance Seasoned Veteran

      Hi everyone,

       

      There seems to be a known limitation where you cannot specific a data profiling row limit when using a JDBC connection. EDC just ignores it. We have run into this when trying to scan a DataBricks connection. For some tables it is fine but we have some that are in excess of  2 Billion rows. We have to limit the number of rows for profiling in this case.

       

      We have confirmed this with Informatica and there is no known fix through version 10.5.1

       

      I am hoping some of you smart people have run into the same issue and might have suggestions for a workaround.

       

      Thanks!

        • 1. Re: JDBC row limit issue
          Srinivas Pai Guru

          Hey Brian

           

           

          This has been fixed in 10.5.1 version of EDC services

          We have added a feature allows the administrator of the resource to add a custom filter/parameter for the resource to be applied while sampling the dataset when using the JDBC scanner:

          ex:

          -DcustomSampling='limit 1000'

           

           

          You can check the release notes for 10.5.1 which mentions this point 

          • Support custom data profile sampling SQL query parameter for the JDBC resource.

           

           

          You should consider upgrading your services to latest version 10.5.1

           

           

          Regards

          Srinivas

          • 2. Re: JDBC row limit issue
            Brian T Vance Seasoned Veteran

            Hi Srinivas,

             

            Thanks for the info. This is good to know.

             

            Is there any chance this will be offered as an EBF for 10.4.1?

             

            Thanks!

            • 3. Re: JDBC row limit issue
              Srinivas Pai Guru

              Hello Brian

               

              I am not sure about the possibility of EBF for the lower versions of services.

              You can raise a GCS ticket who can further check with PM/ENGG on the possibility of down porting the fix to lower versions.

              But from my implementation experience I would suggest you to consider an upgrade of services to latest versions

               

              Regards

              Srinivas