Writing simple Spark DataFrame into MongoDB in nested structure?

Viewed 969

I'm working in pyspark and I want to save simple dataframes in nested JSON like structure into MongoDB. The dataframe has schema like:

root
 |-- Name: string (nullable = true)
 |-- Age: integer (nullable = true)
 |-- City: string (nullable = true)
 |-- Contact number: int (nullable = true)

I used the DataFrameWriter API's write method to save the dataframe in JSON format and I used mongo-spark-connector package:

output_df.write.format("com.mongodb.spark.sql.DefaultSource").mode("append").save();

This is pretty straightforward. I get the MongoDB output for every row in the data frame like:

{
    "Name": "John",
    "Age": 24,
    "City": "Melbourne"
    "Contact number": 123456
}
{
    "Name": "Wauldron",
    "Age": 49,
    "City": "LA"
    "Contact number": 987654
}

They are separate documents. However, I want to save them in a nested structure like:

"Personal Details": [{
    "Name": "John",
    "Age": 24,
    "City": "Melbourne"
    "Contact number": 123456
},
{
    "Name": "Wauldron",
    "Age": 49,
    "City": "LA"
    "Contact number": 987654
}]

I couldn't get this to work. Help me with this.

0 Answers
Related