Python 2.6 JSON decoding performance

James_Austin_51135 0
Viewed 37458

I'm using the json module in Python 2.6 to load and decode JSON files. However I'm currently getting slower than expected performance. I'm using a test case which is 6MB in size and json.loads() is taking 20 seconds.

I thought the json module had some native code to speed up the decoding?

How do I check if this is being used?

As a comparison, I downloaded and installed the python-cjson module, and cjson.decode() is taking 1 second for the same test case.

I'd rather use the JSON module provided with Python 2.6 so that users of my code aren't required to install additional modules.

(I'm developing on Mac OS X, but I getting a similar result on Windows XP.)

python-2.6 python json
7 Answers

It may vary by platform, but the builtin json module is based on simplejson, not including the C speedups. I've found simplejson to be as a fast as python-cjson anyway, so I prefer it since it obviously has the same interface as the builtin.

try:
    import simplejson as json
except ImportError:
    import json

Seems to me that's the best idiom for awhile, yielding the performance when available while being forwards-compatible.

Coady_364330

For those who are parsing output from a request using the requests package, e.g.:

res = requests.request(...)

text = json.loads(res.text)

This can be very slow for larger response contents, say ~45 seconds for 6 MB on my 2017 MacBook. It is not caused by a slow json parser, but instead by a slow character set determination by the res.text call.

You can solve this by setting the character set before you are calling res.text, and using the cchardet package (see also here):

if res.encoding is None:
    res.encoding = cchardet.detect(res.content)['encoding']

This makes the response text json parsing almost instant!

user_34599780

Looking in my installation of Python 2.6.1 on windows, the json package loads the _json module, which is built into the runtime. C source for the json speedups module is here.

>>> import _json
>>> _json
<module '_json' (built-in)>
>>> print _json.__doc__
json speedups
>>> dir(_json)
['__doc__', '__name__', '__package__', 'encode_basestring_ascii', 'scanstring']
>>>
gimel_64910
Related