Read a large zipped text file line by line in python

user_438829 0
Viewed 29919

I am trying to use zipfile module to read a file in an archive. the uncompressed file is ~3GB and the compressed file is 200MB. I don't want them in memory as I process the compressed file line by line. So far I have noticed a memory overuse using the following code:

import zipfile
f = open(...)
z = zipfile.ZipFile(f)
for line in zipfile.open(...).readlines()
  print line

I did it in C# using the SharpZipLib:

var fStream = File.OpenRead("...");
var unzipper = new ICSharpCode.SharpZipLib.Zip.ZipFile(fStream);
var dataStream =  unzipper.GetInputStream(0);

dataStream is uncompressed. I can't seem to find a way to do it in Python. Help will be appreciated.

zip stream python
2 Answers

If the inner directory and the subdirectory filenames in the zipped file don't matter, you can try this:

from zipfile import ZipFile
from io import TextIOWrapper

def zip_open(filename):
    """Wrapper function that for zipfiles."""
    with ZipFile(filename) as zipfin:
        for filename in zipfin.namelist():
            return TextIOWrapper(zipfin.open(filename))

# Usage of the zip_open function)
with zip_open('myzipball.zip') as fin:
    for line in fin:
        print(line)

The zip_open works well when the zipfile contains a single or multiple files without subdirectories. Not sure if the simple for filename in zipfin.namelist() works if there are complex subdirectories structure in the zipped file though.

user_6105690
Related