SBT with Spark-Core 3.0.1 Exception NoClassDefFound for org/apache/log4j/Logger

user_14536848 0
Viewed 387

I am using Log4j with Scala 2.12.12 and Spark-Core 3.0.1 but when I change the library dependencies to not package spark-core in the fat jar I get the following error when I try to run it:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Logger
    at com.some.package.name.Utils$.setup(Utils.scala:207)
    at com.some.package.name.Main$.main(Main.scala:9)
    at com.some.package.name.Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 3 more

The compilation is successful and if I remove the provided clause from the dependencies line everything works fine. My build.sbt is as follows:

scalaVersion := "2.12.12"

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "3.0.1" % "provided",
"org.apache.logging.log4j" % "log4j-api" % "2.13.3",
"org.apache.logging.log4j" % "log4j-core" % "2.13.3",
"org.scalatest" %% "scalatest" % "3.2.0" % "test",
"com.holdenkarau" %% "spark-testing-base" % "3.0.1_1.0.0" % Test)

If I remove my code which writes to the logger the setup for SparkContext is indicated as the line from which the error originates.

apache-spark sbt intellij-idea log4j scala
1 Answers

I am having a similar problem when trying to run my application on EMR 6.2.

I have found a comment on qubole github project claiming AWS' spark-core JAR is missing org/apache/spark/internal/Logging$class, but I have no way to verify if this is true.

user_11655040
Related