Spark SQL's Scala API - TimestampType - No Encoder found for org.apache.spark.sql.types.TimestampType

Viewed 6160

I am using Spark 2.1 with Scala 2.11 on a Databricks notebook

What is exactly TimestampType ?

​We know from ​​SparkSQL's documentation​ that's the official timestamp type is TimestampType, which is apparently an alias for java.sql.Timestamp :

TimestampType can be found here in the​ SparkSQL's Scala ​API

We have a difference when using a schema and the Dataset API

When parsing {"time":1469501297,"action":"Open"} from the Databricks' Scala Structured Streaming example

Using a Json schema --> OK (I do prefer using the elegant Dataset API) :

val jsonSchema = new StructType().add("time", TimestampType).add("action", StringType)

val staticInputDF = 
  spark
    .read
    .schema(jsonSchema)
    .json(inputPath)

Using the Dataset API --> KO: No Encoder found for TimestampType

Creating the Event class

import org.apache.spark.sql.types._
case class Event(action: String, time: TimestampType)
--> defined class Event

Errors when reading the events from DBFS on databricks.

Note: we don't get the error when using java.sql.Timestamp as a type for "time"

val path = "/databricks-datasets/structured-streaming/events/"
val events = spark.read.json(path).as[Event]

Error message

java.lang.UnsupportedOperationException: No Encoder found for org.apache.spark.sql.types.TimestampType
- field (class: "org.apache.spark.sql.types.TimestampType", name: "time")
- root class: 
2 Answers
Related