How to incrementally write into a json file

user_971437 0
Viewed 9146

I am writing a program, which requires me to generate a very large json file. I know the traditional way is to dump a dictionary list using json.dump(), but the list just got too big that even the total memory + swap space cannot hold it before it is dumped. Is there anyway to stream it into a json file, i.e., write the data into the json file incrementally?

dictionary python json
3 Answers

You could also assume you have an iterable it that you want to write out to a file handle fh as a large JSON array of records, and do the following, which I think is the most straightforward approach:

def write_json_iter(it, fh):
    print("[", file=fh)
    for n, rec in enumerate(it):
        if n > 0:
            print(",", file=fh)
        json.dump(rec, fh)
    print("]", file=fh)
user_147581710
Related