How to read data from mongodb atlas from pyspark?

user_8236029 0
Viewed 143

I am trying to read data from mongodb atlas from pyspark but it is throwing an error. Py4JJavaError: An error occurred while calling o204.load.

: java.lang.IllegalArgumentException: requirement failed: Invalid uri: 'mongodb+srv:<url>

I believe this is throwing an Invalid url due to srv along mongodb.

I want to provide jars required for connection using --jars instead --packages as --packages will trigger run time download of required connector .

pyspark mongodb python
1 Answers

With pyspark you can either set up a shell with the proper configuration or set the connection with python code:

from pyspark.sql import SparkSession
import pymongo

if __name__ == "__main__":
    spark = SparkSession.\
builder.\
appName("pyspark-notebook2").\
config("spark.executor.memory", "1g").\
config("spark.mongodb.input.uri","mongodb://127.0.0.1:27017/database.recipes").\
config("spark.mongodb.output.uri","mongodb://127.0.0.1:27017/database.coll").\
config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.12:3.0.0").\
getOrCreate()

    logger = spark._jvm.org.apache.log4j
    logger.LogManager.getRootLogger().setLevel(logger.Level.FATAL)

    
    # read from MongoDB collection
    df = spark.read.format("mongo").load()
user_13652200
Related