Spark submit not working with Protobuf dependency

user_5828621 0
Viewed 1004

Apache Spark: 3.0.0 Protobuf: 3.5.1

Exception: "main" java.lang.NoSuchMethodError: com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List;

Issue: When submitting my spake scala application on my local kubernetes, I am getting: "java.lang.NoSuchMethodError: com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List;"

Seems there is a conflict in dependency for Protobuff.

Have tried a few things like https://github.com/nats-io/stan.java/issues/20 but nothing is working.

my build.sbt:

name := "test"

version := "0.1"
scalaVersion := "2.12.8"

val sparkVersion = "3.0.0"
val protobufVersion = "3.5.1"

resolvers += "confluent" at "http://packages.confluent.io/maven/"
resolvers += Resolver.jcenterRepo
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"

assemblyMergeStrategy in assembly := {
  case "META-INF/services/org.apache.spark.sql.sources.DataSourceRegister" => MergeStrategy.concat
  case PathList("META-INF", xs@_*) => MergeStrategy.discard
  case "application.conf" => MergeStrategy.concat
  case x => MergeStrategy.first
}

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.protobuf.*" -> "shadedproto.@1").inProject
    .inLibrary("com.google.protobuf" % "protobuf-java" % protobufVersion)
)

coverageEnabled.in(ThisBuild, IntegrationTest, test) := true

//skipping test cases during package
test in assembly := {}

lazy val server = (project in file("."))
  .configs(IntegrationTest)
  .settings(Defaults.itSettings)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % sparkVersion % Provided,
  "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion,
  "org.apache.kafka" % "kafka-clients" % "2.2.1",
  "org.postgresql" % "postgresql" % postgreSqlVersion,
  "com.google.protobuf" % "protobuf-java" % protobufVersion,
  "com.typesafe" % "config" % "1.4.0"
)
apache-spark-sql apache-spark kubernetes sbt protocol-buffers
3 Answers

To resolve this you need todo two things. 1) remove the duplicated protobuf library so that there is only one version of it avaible. 2) fix any code that did not use the version of protobuff that spark uses.

One of my favorite quick and dirty tricks to resolve this is to do class search in Intellij of the conflicting class, in this case "Descriptors". This will show all jars that contain the class. Once you've figured out which jars are bringing in the conflicting class, you can remove one of them. Chances are its going to be easiest to just match spark's version of protobuf.

enter image description here

If you sift through the dependencies in maven it seems like it should be using v2.5.0 so some other dependency may be bringing it in.

You may need to check your code to match this version. Whats the full stack trace of your error? Is it your code calling the protobuf function or is it another library? If its another library you may need to fork the library to make it compatible with sparks dependencies.

user_18196180

As this is something that i am currently trying to resolve my self.

In essense spark_parent is using protobuf version 2.5. You are using protobufVersion 3.5 enter image description here Working on Intelij will not have any issues with the following code but when trying to spark-submit the code from the jar file you will encounter the following problem

enter image description here

In essence the spark-parent forces the older version of protobuf.

Excluding from maven, a parent dependancy is not do-able. Recompilining your proto files to 2.5 is out of the questions and will not work. I have tried shading the jar, but havent managed to get that working. Just download spark code to see if there is something that can be done from that part. Will revert back when i have a solution for this

-- Solved : You do not need to create a Fat Jar to get this working. You can use an experimental feature in order to force spark to use your package. Now i am using Spring Boot as well so i do not have to define my package to be executed but this is the spark-submit that is needed

spark-submit  -v --conf spark.driver.userClassPathFirst=true  \
--conf spark.executor.userClassPathFirst=true \
--driver-java-options "-Dspring.profiles.active=local -Djava.util.logging.config.file=log4j.properies " \
--jars /path/to/com/google/protobuf/protobuf-java/3.17.3/protobuf-java-3.17.3.jar \
target/SparkStream-0.1.jar

Hope this helps you out. Do post your solution as well.

user_2996760

Use a shade plugin to shade your version of protobuf. This should resolve the issue.

user_37869570
Related