My code is just to select a double column, but I get floatWritable cannot be cast to DoubleWritable error. Is that because spark has different procedure when reading hive double column?
val testBase = spark.sql(
s"""select
| cast(clk_rate_7_day as double)
|from %s
|where ds between date_sub('%s', 0) and '%s'
|union all
|select
| cast(clk_rate_7_day as double)
|from %s
|where ds between date_sub('%s', 0) and '%s'
|""".stripMargin.format(trainDataInput, jobDate, jobDate, trainDataYuncunInput, jobDate, jobDate))
testBase.show()
but I get this error,
testBase: org.apache.spark.sql.DataFrame = [clk_rate_7_day: double]
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 11, hadoop4445.jd.163.org, executor 5): java.lang.ClassCastException: org.apache.hadoop.io.FloatWritable cannot be cast to org.apache.hadoop.io.DoubleWritable
at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableDoubleObjectInspector.get(WritableDoubleObjectInspector.java:36)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$14$$anonfun$apply$8.apply(TableReader.scala:423)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$14$$anonfun$apply$8.apply(TableReader.scala:423)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:460)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:451)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)