Customize the write operation in Mongo from Spark

user_13567548 0
Viewed 169

How can I write to mongo using spark considering the following scenarios :

  1. If the document is present, just update the matching fields with newer value and if the field is absent, add the new field. (The replaceDocument parameter if false will update the matching records but not add the new unmatched fields while if set to true, my old fields can get lost.)
  2. I want to keep a datafield as READ-ONLY, example there are two fields, first_load_date and updated_on. first_load_date should never change, it is the day that record is created in mongo, and updated_on is when new fields are added or older ones replaced.
  3. If document is absent, insert.

Main problem is replaceDocument = True will lead to loss of older fields not present in newer row, while False, will take care of matched but now the newer incoming fields.

I am using Mongo-Spark-Connector 2.4.1

df.write.format("mongo").mode("append").option("replaceDocument","true").option("database","db1").option("collection","my_collection").save()
pyspark apache-spark-sql apache-spark mongodb-query mongodb
1 Answers

I understood what you are trying to achieve here: You can use something like :

(df
 .write
 .format("mongo")
 .mode("append")
 .option("ordered", "false")
 .option("replaceDocument", "false")
 .option("database", "db1")
 .option("collection", "my_collection")
 .save()
 )

replaceDocument set to false will help you in preserving your old records and updating the matched ones while the you can get a BulkWriteException for which the ordered parameter set to false will help.

user_82067990
Related