How to convert jsonreader object to Python Dataframe?

user_10355900 0
Viewed 1757

I have a large Json file. When i try to read from Pandas as below, it fails with Memory error.

pd.read_json('http://10.10.10.10/rest/get_file.json')

So i tried using Chunk size. Now it returns jsonreader object

pd.read_json('http://10.10.10.10/rest/get_file.json', orient="records", lines=True, chunksize=50) Out[3]: <pandas.io.json.json.JsonReader at 0x1a6a35b0>

How to convert this json reader to Dataframe?

pandas python json
1 Answers

According to the pandas documentation, you can iterate over the Json Reader,


In [262]: jsonl = '''
   .....:     {"a": 1, "b": 2}
   .....:     {"a": 3, "b": 4}
   .....: '''
   .....: 

In [263]: df = pd.read_json(jsonl, lines=True)

In [264]: df
Out[264]: 
   a  b
0  1  2
1  3  4

In [265]: df.to_json(orient='records', lines=True)
Out[265]: '{"a":1,"b":2}\n{"a":3,"b":4}'

# reader is an iterator that returns `chunksize` lines each iteration
In [266]: reader = pd.read_json(StringIO(jsonl), lines=True, chunksize=1)

In [267]: reader
Out[267]: <pandas.io.json._json.JsonReader at 0x7f602de65520>

In [268]: for chunk in reader:
   .....:     print(chunk)
   .....: 
Empty DataFrame
Columns: []
Index: []
   a  b
0  1  2
   a  b
1  3  4

This snapshot is from official documentation https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-json-reader

user_74526270
Related