MongoDB json file to Spark Dataframe

Question

I have a json file extracted from MongoDB with the current format:

{"_id": ObjectId('5112defe123'), "DocumentID": "0001220", "Arrays":[{"id": "1231", "name": "Aaaa"}]}

When I try to load it into a Spark Dataframe and print the schema:

df = spark.read.json('JSON_PATH')
df.printSchema()

I get:

root |-- _corrupt_record: string (nullable = true)

I understand that ObjectId('') is not of a proper Json format and also the json is 1 one line per record. But I was able to do this on python to extract the values and create a pandas dataframe with it later on:

class ObjectId:
    def __init__(self, val):
        self.id = val
.
.
.
s3_file = obj.get()['Body'].read().decode()
for line in s3_file.splitlines():
    ele = eval(line)
    print(ele['_id'].id)

And extract and use the values on that 'Object' as a regular JSON file. It is possible to reproduce that solution on Spark?

MongoDB json file to Spark Dataframe

0 Answers