I'm running into an error when saving data in delta format:
$ data.write.format("delta").save("gs://xxx/delta-table1")
org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)
...
Caused by: java.lang.ArrayStoreException: org.apache.spark.sql.delta.actions.AddFile
at scala.runtime.ScalaRunTime$.array_update(ScalaRunTime.scala:75)
at scala.Array$.slowcopy(Array.scala:84)
at scala.Array$.copy(Array.scala:110)
...
at scala.collection.AbstractTraversable.toArray(Traversable.scala:108)
at org.apache.spark.sql.delta.files.DelayedCommitProtocol.commitJob(DelayedCommitProtocol.scala:59)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:187)
The error occurs only when I use Jupyter with Spylon kernel, if I use spark-shell it just works. Any idea about the possible cause?
In spark-shell:
$ spark-shell --packages io.delta:delta-core_2.12:0.4.0
scala> import io.delta._
import io.delta._
scala> val data = spark.range(0,5)
data: org.apache.spark.sql.Dataset[Long] = [id: bigint]
scala> data.write.format("delta").save("gs://xxx/delta-table1")
scala> val df = spark.read.format("delta").load("gs://xxx/delta-table1")
df: org.apache.spark.sql.DataFrame = [id: bigint]
scala> df.show()
+---+
| id|
+---+
| 2|
| 3|
| 4|
| 0|
| 1|
+---+
In Jupyter with Spylon kernel:
[0] %%init_spark
launcher.packages = [
"io.delta:delta-core_2.12:0.4.0"
]
[1] import io.delta._
val data = spark.range(0,5)
data.write.format("delta").save("gs://xxx/delta-table1")
org.apache.spark.SparkException: Job aborted.
...
Caused by: java.lang.ArrayStoreException: org.apache.spark.sql.delta.actions.AddFile