Python: How to load a JSON string bigger than 1GB

user_8647666 0
Viewed 105

We get a lot of files coming from the eBay API and they are all "1 line JSON string".

MOST(99%) of these files have a size less than 1GB Sometimes 1 of the file will be bigger than 1 GB and in that case the regular json.loads() function that I have used till now will crash the execution of the script and output the simple "Killed" message on the stdout.

For example, this basic line of code

dlist = json.loads(big_string)

will crash. I tried bigjson.load(filename), but bigjson takes over and does not return a JSON object, it will return a bigjson object instead and everything inside that data structure will also become a bigjson object... My algorithm needs dict and list and so I cannot use bigjson.

What can I use to load a JSON String(not file) that is bigger than 1 GB

(I use python version 3.8)

python json
1 Answers

I recently had to work with 8GB of JSON data and I tried using following two solutions and in the end went with second solution

  1. you can try ijson module for loading large files
  2. you can use mongodb to load JSON file and then use pymongo to retrieve data in python it will be in python dict and is similar to JSON and will work with pandas as well
user_70676200
Related