How to read byte string of JSON into pandas

user_9951273 0
Viewed 141

I have a python byte string, that I want to read into pandas. The filetype is NDJSON (newline delimited json) which pandas reads as completely invalid.

s = b'{"id": "01", "amount_usd": 16.44, "foo": 1462075200.0, "bar": 1462075200.0, "baz": "abc"}\n
{"id": "02", "amount_usd": 2.88, "foo": 1462075200.0, "bar": 1462075200.0, "sample": "def"}\n
...

I have tried decoding it and reading it into pandas like so:

s = s.decode("utf-8")
pd.read_json(s)

But this doesn't work because the JSON structure is not what pandas expects.

I get the error

ERROR - Trailing data

How can I read this structure into pandas?

pandas python json
1 Answers

It's true, that's not valid JSON. If they truly are separated by newlines, you can do

s = '[' + ','.join(s.rstrip().split('\n')) + ']'

to turn it into a list.

user_18833160
Related