Compresing large files with gzip in python

user_2228592 0
Viewed 842

I searched how to compress a file in python, and found an answer that was basically as described below:

with open(input_file, 'rb') as f_in, gzip.open(output_file, 'wb') as f_out:
    f_out.write(f_in.read())

It works readily with a 1GB file. But I plan on compressing files up to 200 GB.

Are there any considerations I need to take into account? Is there a different way I should be doing it with large files like that?

The files are binary .img files (exports of a block device; usually with empty space at the end, thus the compression works wonderfully).

gzip python
1 Answers

This will read the entire file into memory, causing problems for you if you don't have 200G available!

You may be able to simply pipe the file through gzip, avoiding Python which will handle doing the work in chunks

% gzip -c myfile.img > myfile.img.gz

Otherwise you should read the file in chunks (picking a large block size may provide some benefit)

BLOCK_SIZE = 8192

with open(myfile, "rb") as f_in, gzip.open(output_file, 'wb') as f_out:
    while True:
        content = f_in.read(BLOCK_SIZE)
        if not content:
            break
        f_out.write(content)
user_45410450
Related