1 Reply Latest reply on Jul 6, 2020 1:34 AM by Smitha HC

    Error while fetching data from Google BigQuery Source System after integrating it with IICS.

    Akshay Anand New Member

      Hi Everyone,

       

      I have an issue while fetching the data from Google BigQuery System and target being the Flat File (CSV).

      Steps that I followed:

      1. Connection mode: Simple
      2. Read mode: Direct ( we do not have the permission to create a google cloud storage bucket for staging files)
      3. Allowed Large Dataset
      4. Source Table being of Size 262 MB
      5. We thought of two condition :
      • We will just use the PreSQL where we will fetch the data for a day using where clause in the source with PreSql Configuration set to USELEGACY:False , Also the table in the BigQuery is partitioned on day basis.

                error: [Querying tables partitioned on a field is not supported in Legacy SQL], using the below KB I am not sure

                if we can get the complete dataset we want in the CSV.

                 KB:550931 Suggested to use Complex or Hybrid Connection mode    

      mapping

      • Secondly, we though of using the filter transformation and using the filter condition to get a day's data:

                     to_date( to_char(*****************,'MM/DD/YYYY HH24:MI:SS'),'MM/DD/YYYY HH24:MI:SS')=to_date(to_char(Systimestamp(),'MM/DD/YYYY HH24:MI:SS'),'MM/DD/YYYY HH24:MI:SS')

      • Then Field mapping, we selected the incoming fields that we wanted.
      • We couldn't find any such information regarding the below error

                    

      ERROR] The following error occurred: [400 Bad Request

       

       

      {

       

       

        "code" : 400,

       

       

        "errors" : [ {

       

       

          "domain" : "global",

       

       

          "message" : "Required parameter is missing",

       

       

          "reason" : "required"

       

       

        } ],

       

       

        "message" : "Required parameter is missing",

       

       

        "status" : "INVALID_ARGUMENT"

      second mapping

      So please let me know if anybody has any suggestion how can we overcome these issue as we have to get the data out of BigQuery System to FF(CSV) using IICS. We do not have any permission on BigQuery system.

       

      Best Regards,

      Akshay Anand