3 Replies Latest reply on Mar 18, 2021 5:25 AM by Premkumar S

    Truncate on Non-Managed Hive table throws exception

    Premkumar S Seasoned Veteran

      Hi All,

       

      We use dynamic mapping and do processing. We are truncating the same target in 2 ways in the same mapping. Due to some earlier issues it was setup like this.

       

      1. Using the truncate target and truncate target partition enabled.

      2. Target PreSQL enabled - Alter external to internal --> Truncate --> Alter internal to external table.

       

      I believe somehow the Truncate target option is getting executed before the Target PreSQL and removing the file in S3. Once the files are cleared in S3, the folder disappears as it is empty.

       

      Now the PreSQL executes and this is throwing an "FileNotFoundException" as below.

       

      WARNING: java.sql.SQLException: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing

      at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)

      at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257)

      at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)

      at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)

      at java.security.AccessController.doPrivileged(Native Method)

      at javax.security.auth.Subject.doAs(Subject.java:422)

      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)

      at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)

      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Exception while processing

      at org.apache.hadoop.hive.ql.exec.DDLTask.truncateTable(DDLTask.java:4672)

      at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:563)

      at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:201)

      at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)

      at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)

      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)

      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)

      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)

      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1232)

      at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:255)

      ... 11 more

      Caused by: java.io.FileNotFoundException: No such file or directory: s3a://buckename/staging/dummy

      at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:996)

      at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)

      at org.apache.hadoop.hive.io.HdfsUtils$HadoopFileStatus.<init>(HdfsUtils.java:268)

      at org.apache.hadoop.hive.ql.exec.DDLTask.truncateTable(DDLTask.java:4654)

      ... 20 more

       

      I have below questions:

      1. Is the truncate target from mapping running before Target PreSQL.

      2. Or is this an access issue? Because of the highlighted error message.

       

      We are using mapping impersonation. Table owner is X and Mapping Impersonation name is Y.

       

      Can this be an issue?