I'm a beginner of scala/spark and I got stuck when shipping my code to the official environment.
To be short, I can't put my SparkSession object in class method and I don't know why? If I do so, it will be fine when I run it on a local single machine but throw java.lang.NoClassDefFoundError, Could not initialize class XXX when I package my code to a single jar file and run it on multiple machines using spark-submit.
For example
When I put my code in structure like this
object Main{
def main(...){
Task.start
}
}
object Task{
case class Data(name:String, ...)
val spark = SparkSession.builder().appName("Task").getOrCreate()
import spark.implicits._
def start(){
var ds = loadFile(path)
ds.map(someMethod) // it dies here!
}
def loadFile(path:String){
spark.read.schema(...).json(path).as[Data]
}
def someMethod(d:Data):String{
d.name
}
}
It will give me "java.lang.NoClassDefFoundError" on each places where I put a self-defined method in those dataset transformation functions (like map, filter... etc).
However, if I rewrite it as
object Task{
case class Data(name:String, ...)
def start(){
val spark = SparkSession.builder().appName("Task").getOrCreate()
import spark.implicits._
var ds = loadFile(spark, path)
ds.map(someMethod) // it works!
}
def loadFile(spark:SparkSession, path:String){
import spark.implicits._
spark.read.schema(...).json(path).as[Data]
}
def someMethod(d:Data):String{
d.name
}
}
It will be fine, but it means that I need to pass the "spark" variable through each of methods that I will need it and I need to write import spark.implicits._ all the time when a method need it.
I think something goes wrong when the spark try to shuffle my object between nodes, but I don't know how exactly the reason is and what is the correct way to write my code.
Thanks