How I can I lazily read multiple JSON values from a file/stream in Python?

user_1114 0
Viewed 91245

I'd like to read multiple JSON objects from a file/stream in Python, one at a time. Unfortunately json.load() just .read()s until end-of-file; there doesn't seem to be any way to use it to read a single object or to lazily iterate over the objects.

Is there any way to do this? Using the standard library would be ideal, but if there's a third-party library I'd use that instead.

At the moment I'm putting each object on a separate line and using json.loads(f.readline()), but I would really prefer not to need to do this.

Example Use

example.py

import my_json as json
import sys

for o in json.iterload(sys.stdin):
    print("Working on a", type(o))

in.txt

{"foo": ["bar", "baz"]} 1 2 [] 4 5 6

example session

$ python3.2 example.py < in.txt
Working on a dict
Working on a int
Working on a int
Working on a list
Working on a int
Working on a int
Working on a int
serialization python json
12 Answers

You can use https://pypi.org/project/json-stream-parser/ for exactly that purpose.

import sys
from json_stream_parser import load_iter
for obj in load_iter(sys.stdin):
    print(obj)

output

{'foo': ['bar', 'baz']}
1
2
[]
4
5
6
user_38868990

If you have control over how the data is generated, you may want to switch to an alternate format like ndjson, which stands for Newline Delimited JSON and allows to stream incremental JSON-formatted data. Each line is valid JSON on its own. There are two Python packages available: ndjson and jsonlines.

There is also json-stream which allows you to process JSON as you read it, thus avoiding having to load the entire JSON upfront. You should be able to use it to read JSON data from a stream, but also use the same stream before and after for any other I/O operations, or to read several JSON objects from the same stream.

user_4243810
Related