I'm working in pyspark and I want to save simple dataframes in nested JSON like structure into MongoDB. The dataframe has schema like:
root
|-- Name: string (nullable = true)
|-- Age: integer (nullable = true)
|-- City: string (nullable = true)
|-- Contact number: int (nullable = true)
I used the DataFrameWriter API's write method to save the dataframe in JSON format and I used mongo-spark-connector package:
output_df.write.format("com.mongodb.spark.sql.DefaultSource").mode("append").save();
This is pretty straightforward. I get the MongoDB output for every row in the data frame like:
{
"Name": "John",
"Age": 24,
"City": "Melbourne"
"Contact number": 123456
}
{
"Name": "Wauldron",
"Age": 49,
"City": "LA"
"Contact number": 987654
}
They are separate documents. However, I want to save them in a nested structure like:
"Personal Details": [{
"Name": "John",
"Age": 24,
"City": "Melbourne"
"Contact number": 123456
},
{
"Name": "Wauldron",
"Age": 49,
"City": "LA"
"Contact number": 987654
}]
I couldn't get this to work. Help me with this.