Exception in thread "main" java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)

user_6121345 0
Viewed 27730

Any reason why I get this error ? Initially the IDE plugin for Scala was 2.12.3. But since I'm working with Spark 2.2.0, I manually changed it to Scala 2.11.11.

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/09/19 12:08:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
    at scala.xml.Null$.<init>(Null.scala:23)
    at scala.xml.Null$.<clinit>(Null.scala)
    at org.apache.spark.ui.jobs.AllJobsPage.<init>(AllJobsPage.scala:39)
    at org.apache.spark.ui.jobs.JobsTab.<init>(JobsTab.scala:38)
    at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:67)
    at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:84)
    at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:221)
    at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:163)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:452)
    at sparkEnvironment$.<init>(Ticket.scala:33)
    at sparkEnvironment$.<clinit>(Ticket.scala)
    at Ticket$.main(Ticket.scala:39)
    at Ticket.main(Ticket.scala)
apache-spark sbt intellij-idea scala
3 Answers

Scala Version 2.12 gave me the similar error, however after removing the extends App and adding the main method, everything is working fine.

object SparkDemo extends App{}

Just remove extends App and add main function

object SparkDemo {
def main(args: Array[String]): Unit = {}
}

Spark Document itself even suggests using main function rather than extends App.

https://spark.apache.org/docs/2.4.0/quick-start.html#:~:text=Note%20that%20applications%20should%20define%20a%20main()%20method%20instead%20of%20extending%20scala.App.%20Subclasses%20of%20scala.App%20may%20not%20work%20correctly.

This problem could be caused by any dependency as well; If a dependency was internally compiled with Scala 2.11, you will need to downgrade your version of Scala.

user_121203010

Its a spark dependency issue. Scala 2.12 compatible with spark 3 & scala 2.11 compatible with spark 2

user_15639690
Related