When I am running the scala code from Intellij in local mode, it works fine. However, when I am trying to run it in cluster mode, I am getting the following error.
I even tried creating a jar and used Spark-Submit, even then I am getting a separate error.
var connectionString = MongoDBCredentials.getConnectionString(view_name)
spark.read.format("com.mongodb.spark.sql.DefaultSource").
option("spark.mongodb.input.partitioner", "MongoSinglePartitioner").
option("uri", connectionString).
load()
Caused by: java.lang.ClassNotFoundException: com.mongodb.spark.rdd.partitioner.MongoPartition
It looks like my MongoDB dependency is somehow not going over to executors. But how should I get it pushed to executors?
I am using the Spark cluster running in docker container. Here is my code that I am using Github
After reading few articles, I tried including the following dependencies in my build.sbt as well but with no luck.
libraryDependencies += "org.mongodb" % "mongodb-driver" % "3.0.1" from "https://repo1.maven.org/maven2/org/mongodb/mongodb-driver/mongodb-driver-3.0.1.jar"
libraryDependencies += "org.mongodb" % "mongodb-driver-core" % "3.8.1" from "https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-core/3.8.1/mongodb-driver-core-3.8.1.jar"
libraryDependencies += "org.mongodb" % "bson" % "3.8.1" from "https://repo1.maven.org/maven2/org/mongodb/bson/3.8.1/bson-3.8.1.jar"
Also added mongodb connector in SparkSession
val spark = SparkSession.builder().config(sc.getConf).
appName("Spark SQL basic example").
config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.12:3.0.1").
config("spark.some.config.option", "some-value").
getOrCreate()