Getting ClassNotFoundException in MongoPartition in Spark in cluster mode, in local mode, code runs fine. How to resolve?

Viewed 208

When I am running the scala code from Intellij in local mode, it works fine. However, when I am trying to run it in cluster mode, I am getting the following error. I even tried creating a jar and used Spark-Submit, even then I am getting a separate error.

var connectionString = MongoDBCredentials.getConnectionString(view_name)
    spark.read.format("com.mongodb.spark.sql.DefaultSource").
    option("spark.mongodb.input.partitioner", "MongoSinglePartitioner").
    option("uri", connectionString).
    load()

Caused by: java.lang.ClassNotFoundException: com.mongodb.spark.rdd.partitioner.MongoPartition

It looks like my MongoDB dependency is somehow not going over to executors. But how should I get it pushed to executors?

I am using the Spark cluster running in docker container. Here is my code that I am using Github

After reading few articles, I tried including the following dependencies in my build.sbt as well but with no luck.

libraryDependencies += "org.mongodb" % "mongodb-driver" % "3.0.1" from "https://repo1.maven.org/maven2/org/mongodb/mongodb-driver/mongodb-driver-3.0.1.jar"
libraryDependencies += "org.mongodb" % "mongodb-driver-core" % "3.8.1" from "https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-core/3.8.1/mongodb-driver-core-3.8.1.jar"
libraryDependencies += "org.mongodb" % "bson" % "3.8.1" from "https://repo1.maven.org/maven2/org/mongodb/bson/3.8.1/bson-3.8.1.jar"

Also added mongodb connector in SparkSession

val spark = SparkSession.builder().config(sc.getConf).
      appName("Spark SQL basic example").
      config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.12:3.0.1").
      config("spark.some.config.option", "some-value").
      getOrCreate()
0 Answers
Related