Spark Session connected to local MongoDB with pyspark

Question

Apparently simple objective: to create a spark session connected to local MongoDB using pyspark. According to literature, it is only necessary to include mongo's uris in the configuration (mydb and coll exist at mongodb://127.0.0.1:27017):

from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").appName("myApp") \
    .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/mydb.coll") \
    .config("spark.mongodb.output.uri", "mongodb://127.0.0.1/mydb.coll") \
    .getOrCreate()

and to check that the session is operative, we can use something like:

df = spark.read.format("mongo").load()
df.printSchema()

The session is generated without problem, but the result of trying to read is:

java.lang.ClassNotFoundException: Failed to find data source: mongo

I am using an environment in PyCharm with Python 3.8.8, pyspark 3.1.2 and mongo 5.0.2. A first search for solutions leads to information about the necessity to add the following element to the configuration (spark-mongo connector):

.config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector_2.12:3.0.1')\

So that the final instance is:

spark = SparkSession.builder.master("local[*]").appName("myApp") \
    .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/mydb.coll") \
    .config("spark.mongodb.output.uri", "mongodb://127.0.0.1/mydb.coll") \
    .config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector_2.12:3.0.1') \
    .getOrCreate()

But the error remains the same: java.lang.ClassNotFoundException:Failed to find data source: com.mongodb.spark.sql.DefaultSource

Looking for alternatives leads to complex issues like Error connecting to mongodb with mongo-spark-connector which right now fall out my reach. What is so complex about something that looks so straightforward as connecting to mongo with a spark session using pyspark? Isn't there a standard way to do so for the obvious and most simple scenario (pyspark+local mongo)?

Spark Session connected to local MongoDB with pyspark

0 Answers