Persistence & Serialization of Scala case-classes on a cluster

I am looking for a robust, backwards-compatible, serialization & persistence solution for my case-classes.

General

• my project is written in Scala.
• It contains various case-classes that need to be persisted to the disk, and loaded from there when necessary.
• Every version of the project yields a new jar file, and I execute it via spark-submit command.

Today's Problem - case-classes are saved as objects with spark. It is not robust - sometimes when the running environment changes, or if I add a comment and re-make the jar, the serialization is broken and I cannot load my "old" case-classes.

Requirements

The main goal is to be able to save and load case classes both on a cluster and locally, with focus on:

• Robustness - changing environment / re-assembly the jar / variables (NOT arguments) in the case-class, shouldn't affect the serialization and the ability to load "old" objects.
• Efficiency - In term of (1) space, (2) time
• Backwards-Compatible solution (as much/easy to-do as possible)
• Intuitive & Elegant solution, if possible

Current Solution Example

Basically dumping the case-class to the disk:

def writeObject[T: ClassTag](path: String, content: T): Unit ={
    removeDir(new Path(path))
    spark.sparkContext.parallelize(Seq(content), 1).saveAsObjectFile(path)
}

def readObject[T: ClassTag](path: String): Option[T] = {
    Some(spark.sparkContext.objectFile[T](path).first())
}

upickle

I heard about that library. I started checking it out - but not sure how it will work. My main concerns are "complicated" case-classes (Lots of inheritance & traits extending), the need for companion objects for every case-class, lots of implicit values, and counting on macros.

sketch of the solution with upickle:

//    convert to json & save
val jsonString: String = upickle.default.write(caseClassInstance)
spark.sparkContext.parallelize(Seq(jsonString)).saveAsTextFile(path)

//    load json & convert back to case-class
val loadedJsonString: String = spark.sparkContext.textFile(path).take(1)(0)
val caseClassInstance: T = upickle.default.read[T](loadedJsonString)

Edit: This solution (on it's base form) does not serialize inner-structures / variables of my case-classes, but only the arguments..

I would like to hear your thoughts and maybe find a better solution. Thank you!