Get "java.lang.NoClassDefFoundError" when running spark project with spark-submit on multiple machines

Viewed 1124

I'm a beginner of scala/spark and I got stuck when shipping my code to the official environment.

To be short, I can't put my SparkSession object in class method and I don't know why? If I do so, it will be fine when I run it on a local single machine but throw java.lang.NoClassDefFoundError, Could not initialize class XXX when I package my code to a single jar file and run it on multiple machines using spark-submit.

For example

When I put my code in structure like this

object Main{
    def main(...){
        Task.start
    }
} 

object Task{
    case class Data(name:String, ...)
    val spark = SparkSession.builder().appName("Task").getOrCreate()
    import spark.implicits._

    def start(){
        var ds = loadFile(path) 
        ds.map(someMethod) // it dies here!
    }

    def loadFile(path:String){
        spark.read.schema(...).json(path).as[Data]
    }

    def someMethod(d:Data):String{
        d.name
    }
}

It will give me "java.lang.NoClassDefFoundError" on each places where I put a self-defined method in those dataset transformation functions (like map, filter... etc).

However, if I rewrite it as

object Task{
    case class Data(name:String, ...)

    def start(){
        val spark = SparkSession.builder().appName("Task").getOrCreate()
        import spark.implicits._
        var ds = loadFile(spark, path) 
        ds.map(someMethod) // it works!
    }

    def loadFile(spark:SparkSession, path:String){
        import spark.implicits._
        spark.read.schema(...).json(path).as[Data]
    }

    def someMethod(d:Data):String{
        d.name
    }
}

It will be fine, but it means that I need to pass the "spark" variable through each of methods that I will need it and I need to write import spark.implicits._ all the time when a method need it.

I think something goes wrong when the spark try to shuffle my object between nodes, but I don't know how exactly the reason is and what is the correct way to write my code.

Thanks

1 Answers
Related