Spark build.sbt file versioning

Question

I am having a hard time understanding the multiple version numbers going into the build.sbt file for spark programs.

1. version
2. scalaVersion
3. spark version?
4. revision number.

There are multiple compatibility between these versions as well. Can you please explain how to decide these versions for my project.

user_4111404 · Answer

I hope the following SBT lines and their comments will be sufficient to explain your question.

// The version of your project itself.
// You can change this value whenever you want,
// e.g. everytime you make a production release.
version := "0.1.0"

// The Scala version your project uses for compile.
// If you use spark, you can only use a 2.11.x version.
// Also, because Spark includes its own Scala in runtime
// I recommend you use the same one;
//you can check which one your Spark instance uses in the spark-shell.
scalaVersion := "2.11.12"

// The spark version the project uses for compile.
// Because you wont generate an uber jar with Spark included,
// but deploy your jar to an spark cluster instance.
// This version must match with the remote one, unless you want weird bugs...
val SparkVersion = "2.3.1"
// Note, I use a val with the Spark version
// to make it easier to include several Spark modules in my project,
// this way, if I want/have to change the Spark version,
// I only have to modify one line,
// and avoid strange erros because I changed some versions, but not others.
// Also note the 'Provided' modifier at the end,
// it indicates SBT that it shouldn't include the Spark bits in the generated jar
// neither in package nor assembly tasks.
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % SparkVersion % Provided,
  "org.apache.spark" %% "spark-sql" % SparkVersion % Provided,
)

// Exclude Scala from the assembly jar, because spark already includes it.
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

You should also take care of the SBT version, that is the version of the SBT used in your project. You set it in the "project/build.properties" file.

sbt.version=1.2.3

Note: I use the sbt-assembly plugin, to generate a jar with all dependencies included except Spark and Scala. This is usefull if you use other libraries like the MongoSparkConnector for example.

Spark build.sbt file versioning

1 Answers