if I am interpreting this exception well, when I'm using Zeppelin to execute a Spark statement written in Scala, it fails with a ClassCastException involving an $iw class:
- in loader
org.apache.spark.repl.ExecutorClassLoader - against one coming from loader:
scala.tools.nsc.interpreter.IMain$TranslatingClassLoader
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3) (192.168.1.153 executor driver): java.lang.ClassCastException: class $iw cannot be cast to class $iw ($iw is in unnamed module of loader org.apache.spark.repl.ExecutorClassLoader @69bd1241; $iw is in unnamed module of loader scala.tools.nsc.interpreter.IMain$TranslatingClassLoader @5a68e587)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.deserializetoobject_doConsume_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
These two cells do execute correctly if I'm using spark-shell from a command line:
scala> :paste
// Entering paste mode (ctrl-D to finish)
import spark.implicits._
case class Flight(DEST_COUNTRY_NAME: String,
ORIGIN_COUNTRY_NAME: String,
count: BigInt)
val flightsDF = spark.read
.parquet("/home/lebihan/dev/apprentissageDev/Spark-The-Definitive-Guide/data/flight-data/parquet/2010-summary.parquet/")
val flights = flightsDF.as[Flight]
// Exiting paste mode, now interpreting.
import spark.implicits._
defined class Flight
flightsDF: org.apache.spark.sql.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field]
flights: org.apache.spark.sql.Dataset[Flight] = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field]
scala> :paste
// Entering paste mode (ctrl-D to finish)
flights
.filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME != "Canada")
.map(flight_row => flight_row)
.take(5)
flights
.take(5)
.filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME != "Canada")
.map(fr => Flight(fr.DEST_COUNTRY_NAME, fr.ORIGIN_COUNTRY_NAME, fr.count + 5))
// Exiting paste mode, now interpreting.
res7: Array[Flight] = Array(Flight(United States,Romania,6), Flight(United States,Ireland,269), Flight(United States,India,74), Flight(Egypt,United States,29), Flight(Equatorial Guinea,United States,6))
So I believe I'm stumbling upon some kind of configuration mess, in Zeppelin.
But what should I check to resolve it?
My spark interpreter has for SPARK_HOME, /opt/spark-3.1.1-bin-hadoop2.7,
but I don't see anything about a Scala version property to control anywhere, if it's what I shall check...
What am I in front of?