Error saving data in `delta` format in Jupyter with Spylon kernel

Viewed 371

I'm running into an error when saving data in delta format:

$ data.write.format("delta").save("gs://xxx/delta-table1")

org.apache.spark.SparkException: Job aborted.
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)  
...
Caused by: java.lang.ArrayStoreException: org.apache.spark.sql.delta.actions.AddFile
  at scala.runtime.ScalaRunTime$.array_update(ScalaRunTime.scala:75)
  at scala.Array$.slowcopy(Array.scala:84)
  at scala.Array$.copy(Array.scala:110)
  ...
  at scala.collection.AbstractTraversable.toArray(Traversable.scala:108)
  at org.apache.spark.sql.delta.files.DelayedCommitProtocol.commitJob(DelayedCommitProtocol.scala:59)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:187)

The error occurs only when I use Jupyter with Spylon kernel, if I use spark-shell it just works. Any idea about the possible cause?

In spark-shell:

$ spark-shell --packages io.delta:delta-core_2.12:0.4.0

scala> import io.delta._
import io.delta._

scala> val data = spark.range(0,5)
data: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> data.write.format("delta").save("gs://xxx/delta-table1")
                                                                                
scala> val df = spark.read.format("delta").load("gs://xxx/delta-table1")
df: org.apache.spark.sql.DataFrame = [id: bigint]

scala> df.show()
+---+                                                                           
| id|
+---+
|  2|
|  3|
|  4|
|  0|
|  1|
+---+

In Jupyter with Spylon kernel:

[0] %%init_spark

launcher.packages = [

"io.delta:delta-core_2.12:0.4.0"

]

[1] import io.delta._

val data = spark.range(0,5)

data.write.format("delta").save("gs://xxx/delta-table1")

org.apache.spark.SparkException: Job aborted.
...
Caused by: java.lang.ArrayStoreException: org.apache.spark.sql.delta.actions.AddFile

1 Answers

As I see, the spylon kernel is compiled only against Scala 2.11 & Spark 2.x. You're using Delta version compiled against Scala 2.12 (see the _2.12 in the package name), and as result, it's incompatible with spylon. You need to downgrade your Spark to the version compiled with Scala 2.11, like 2.4.7, then you can use it with Spylon.

P.S. Also, it makes sense to use newer version of Delta - 0.4.0 is quite old. The latest compatible with Scala 2.11 is 0.6.1...

Related