Spark on AWS EMR: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

user_1554386 0
Viewed 717

I'm trying to deploy a Scala 2.12.0 + Spark 2.4.4 application to AWS EMR. I have Spark 2.4.4 enabled in EMR.

When the application runs, it fails with the following error:

java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

I'm building the jar using the "sbt-assembly" plugin. I specifically remove "org/apache/spark/" from the assembled jar to ensure no conflict in versions.

amazon-emr apache-spark scala
1 Answers

While Spark 2.4.4 supposedly supports Scala 2.12.x, EMR doesn't seem to support it.

Instead, one must build using Scala 2.11.

E.g. in the build.sbt file:

scalaVersion := "2.11.12"
user_15543860
Related