3 Replies Latest reply on Oct 4, 2021 12:31 PM by Srinivas Pai

    JDBC row limit issue

    Brian T Vance Guru

      Hi everyone,


      There seems to be a known limitation where you cannot specific a data profiling row limit when using a JDBC connection. EDC just ignores it. We have run into this when trying to scan a DataBricks connection. For some tables it is fine but we have some that are in excess of  2 Billion rows. We have to limit the number of rows for profiling in this case.


      We have confirmed this with Informatica and there is no known fix through version 10.5.1


      I am hoping some of you smart people have run into the same issue and might have suggestions for a workaround.



        • 1. Re: JDBC row limit issue
          Srinivas Pai Guru

          Hey Brian



          This has been fixed in 10.5.1 version of EDC services

          We have added a feature allows the administrator of the resource to add a custom filter/parameter for the resource to be applied while sampling the dataset when using the JDBC scanner:


          -DcustomSampling='limit 1000'



          You can check the release notes for 10.5.1 which mentions this point 

          • Support custom data profile sampling SQL query parameter for the JDBC resource.



          You should consider upgrading your services to latest version 10.5.1





          • 2. Re: JDBC row limit issue
            Brian T Vance Guru

            Hi Srinivas,


            Thanks for the info. This is good to know.


            Is there any chance this will be offered as an EBF for 10.4.1?



            • 3. Re: JDBC row limit issue
              Srinivas Pai Guru

              Hello Brian


              I am not sure about the possibility of EBF for the lower versions of services.

              You can raise a GCS ticket who can further check with PM/ENGG on the possibility of down porting the fix to lower versions.

              But from my implementation experience I would suggest you to consider an upgrade of services to latest versions