I'm moving data from one hive table to another using Spark after applying some transformations. While doing so, I'm getting an exception saying avsc file can't be read (as per my understanding) but I can see the avsc file in HDFS.
Please advise what could be the reason.
I've source hive table defined as:
+----------------------------------------------------+--+
| createtab_stmt |
+----------------------------------------------------+--+
| CREATE TABLE `exchr`( |
| `exchr_sk` bigint, |
| `rec_ctry_cd` string, |
| `cob_dt` date, |
| `ccy_cd` string, |
| `exchr_val` decimal(10,5)) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.mapred.TextInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
| LOCATION |
| 'hdfs://xxxx/EXCHR' |
| TBLPROPERTIES ( |
| 'COLUMN_STATS_ACCURATE'='true', |
| 'last_modified_by'='hive', |
| 'last_modified_time'='1500408192', |
| 'numFiles'='0', |
| 'numRows'='3', |
| 'rawDataSize'='73', |
| 'totalSize'='0', |
| 'transient_lastDdlTime'='1501780655') |
+----------------------------------------------------+--+
I've a target hive table defined as:
+----------------------------------------------------+--+
| createtab_stmt |
+----------------------------------------------------+--+
| CREATE EXTERNAL TABLE `fx_rate`( |
| `stamp` string COMMENT '', |
| `curr_code` string COMMENT '', |
| `fx_rate` double COMMENT '') |
| PARTITIONED BY ( |
| `cb_dt` string) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' |
| LOCATION |
| 'hdfs://xxxx/fx_rate' |
| TBLPROPERTIES ( |
| 'avro.schema.url'='hdfs://xxxx/fx_rate.avsc', |
| 'transient_lastDdlTime'='1502313078') |
+----------------------------------------------------+--+
when i'm trying to insert data into this table using spark with below code, I encounter Encountered exception determining schema. Returning signal schema to indicate problem
Spark code:
val TABLE_2_0 = sqlContext.sql("select * from xxxx.exchr");
val SELECT_1_0 = TABLE_2_0.select(TABLE_2_0.col("*"));
val SELECT_0_0 = SELECT_1_0.select(date_format(SELECT_1_0.col("cob_dt"), "yyyymmdd").as("stamp"), lit(null).as("curr_code"), round(lit(10)/SELECT_1_0.col("exchr_val"),10).as("fx_rate"), date_format(SELECT_1_0.col("cb_dt"), "yyyymmdd").as("cb_dt")).limit(1000);
SELECT_0_0.toDF().write.mode("append").insertInto("xxxx.fx_rate")
Exception trace:
java.lang.NullPointerException
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:381)
at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:154)
at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:135)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:172)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:103)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.newSerializer(InsertIntoHiveTable.scala:59)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.outputClass$lzycompute(InsertIntoHiveTable.scala:53)
17/08/10 23:46:44 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 3, myhost.net, executor 2): org.apache.hadoop.hive.serde2.SerDeException: Encountered exception determining schema. Returning signal schema to indicate problem: null
at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:523)
at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:97)
at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:88)
at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:81)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1(InsertIntoHiveTable.scala:92)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:84)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:84)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)