I'm migrating some code from Spark 1.6 to Spark 2.1 and struggling with the following issue:
This worked perfectly in Spark 1.6
import org.apache.spark.sql.types.{LongType, StructField, StructType}
val schema = StructType(Seq(StructField("i", LongType,nullable=true)))
val rows = sparkContext.parallelize(Seq(Row(Some(1L))))
sqlContext.createDataFrame(rows,schema).show
The same code in Spark 2.1.1:
import org.apache.spark.sql.types.{FloatType, LongType, StructField, StructType}
val schema = StructType(Seq(StructField("i", LongType,nullable=true)))
val rows = ss.sparkContext.parallelize(Seq(Row(Some(1L))))
ss.createDataFrame(rows,schema).show
gives the following Runtime exception:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 8.0 failed 4 times, most recent failure: Lost task 0.3 in stage 8.0 (TID 72, i89203.sbb.ch, executor 9): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: scala.Some is not a valid external type for schema of bigint
So how should I translate such code to Spark 2.x if I want to have nullable Long's rather than using Option[Long]?