0 Replies Latest reply on Oct 8, 2019 6:19 AM by Indhumathi Kannan

    Mass Ingestion - Incremental Load

    Indhumathi Kannan New Member

      Hi,

      I'm new to Informatica Mass Ingestion service and using Informatica 10.2.2  hotfix. I tried the Incremental load but ingestion fails.

      Steps I followed :

      1. Ingested a file in full load mode

           Source : Splice

           Target : Hive

           *Successful ingestion

      2. Update the source table by appending new rows

           (I just added few rows with key > previous keys)

      3. Edit the Ingestion specification by enabling "Incremental load"

           * Using ID to identify incremental load column

      4. Redeployed specification

      5. Ran the specification in Incremental mode

      Issue: Error occurs/execution fails whenever I check the incremental load option

      Execution failed with following log:

      2019-10-08 07:40:07.988 <LdtmWorkflowTask-pool-5-thread-3> SEVERE: Spark task [InfaSpark0] failed with condition [com.informatica.sdk.dtm.ExecutionException: [[SPARK_1003] Spark task [InfaSpark0] failed with condition [User class threw exception: java.lang.reflect.InvocationTargetException

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

      at java.lang.reflect.Method.invoke(Method.java:498)

      at com.informatica.compiler.InfaSparkMain$.main(InfaSparkMain.scala:124)

      at com.informatica.compiler.InfaSparkMain.main(InfaSparkMain.scala)

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

      at java.lang.reflect.Method.invoke(Method.java:498)

      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)

      Caused by: java.sql.SQLSyntaxErrorException: Lexical error at line 1, column 1795.  Encountered: "$" (36), after : "".

      at com.splicemachine.db.client.am.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:89)

      at com.splicemachine.db.client.am.SqlException.getSQLException(SqlException.java:368)

      at com.splicemachine.db.client.am.Connection.prepareStatement(Connection.java:413)

      at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:58)

      at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:115)

      at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:52)

      at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)

      at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)

      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)

      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)

      at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:254)

      at com.informatica.bootstrap.functions$.readJdbc(functions.scala:1899)

      at com.informatica.exec.InfaSpark0$$anonfun$1.apply(InfaSpark0.scala:52)

      at com.informatica.exec.InfaSpark0$$anonfun$1.apply(InfaSpark0.scala:52)

      at com.informatica.bootstrap.Step.v$lzycompute(async.scala:110)

      at com.informatica.bootstrap.Step.v(async.scala:110)

      at com.informatica.bootstrap.Step.get(async.scala:111)

      at com.informatica.exec.InfaSpark0$$anonfun$2.apply(InfaSpark0.scala:53)

      at com.informatica.exec.InfaSpark0$$anonfun$2.apply(InfaSpark0.scala:53)

      at com.informatica.bootstrap.Step.v$lzycompute(async.scala:110)

      at com.informatica.bootstrap.Step.v(async.scala:110)

      at com.informatica.bootstrap.Step.get(async.scala:111)

      at com.informatica.exec.InfaSpark0$.main(InfaSpark0.scala:55)

      at com.informatica.exec.InfaSpark0.main(InfaSpark0.scala)

      ... 11 more

       

      Thanks in advance!