decompressing huge json gzip file causing memory error in python

user_2259219 0
Viewed 1540

I got the following problem. I am working with small machine with low memory (1 GB). my program download a huge gzip file from some url. And I need to decompress it to dict I know for sure that the file is in json format. My problem is that after I run the following command I got a memory error:

data = zlib.decompress(url, 16 + zlib.MAX_WBITS).decode('utf8')
results_list.append(json.loads(data ))

now for small files this works fine, but for large I got the error. my intuition tells me that I should split the file into chunks, but then because I am expecting a json file i wont be able to restore the chunks back to json (because each part wont be a valid json string).

what I should do?

Thank a lot!

zlib gzip python-3.x json
1 Answers

Create a decompression object using z=zlib.decompressobj(), and then do z.decompress(some_compressed_data, max), which will return no more than max bytes of uncompressed data. You then call again with z.decompress(z.unconsumed_tail, max) until the rest of some_compressed_data is consumed, and then feed it more compressed data.

You will need to then be able to process the resulting uncompressed data a chunk at a time.

user_11806200
Related