Downloading & extracting in parallel, maximizing performance?

Question

I want to download and extract 100 tar.gz files that are each 1GB in size. Currently, I've sped it up with multithreading and by avoiding disk IO via in-memory byte streams, but can anyone show me how to make this faster (just for curiosity's sake)?

from bs4 import BeautifulSoup
import requests
import tarfile

import multiprocessing as mp
from concurrent.futures import ThreadPoolExecutor

# speed up by only extracting what we need
def select(members):
    for file in members:  
        if any(ext in file.name for ext in [".tif", ".img"]):
            yield file

# for each url download the tar.gz and extract the necessary files
def download_and_extract(x):
    # read and unzip as a byte stream
    r = requests.get(x, stream=True)
    tar = tarfile.open(fileobj=r.raw, mode='r|gz')
    tar.extractall(members=select(tar))
    tar.close()


# parallel download and extract the 96 1GB tar.gz files
links = get_asset_links()
# 3 * cpu count seemed to be fastest on a 4 core cpu
with ThreadPoolExecutor(3 * mp.cpu_count()) as executor:
    executor.map(download_and_extract, links)

My current approach takes 20 - 30 minutes. I'm not sure what the theoretical possible speed up is, but if its helpful, the download speed for a single file is 20 MB/s in isolation.

If anyone could indulge my curiosity, that would be greatly appreciated! Some things I looked into were asyncio, aiohttp, and aiomultiprocess, io.BytesIO etc. But I wasn't able to get them to work well with the tarfile library.

user_12939557 · Answer

Your computation is likely IO bound. Compression is generally a slow task, especially the gzip algorithm (new algorithms can be much faster). From the provided information, the average reading speed is about 70 Mo/s. This means that the storage throughput is at least roughly 140 Mo/s. It looks like totally normal and expected. This is especially true if you use a HDD or a slow SSD.

Besides this, it seems you iterate over the files twice due to the selection of members. Keep in mind that tar gz files are a big block of files packed together and then compressed with gzip. To iterate over the filenames the tar file need to be already partially decompressed. This may not be a problem regarding the implementation of tarfile (possible caching). If the size of all the discarded files is small, it may be better to simply decompress the whole archive in a raw and then remove the files to discard. Moreover, if you have a lot a memory and the size of all discarded files is not small, you can decompress the files in an in-memory virtual storage device first in order to write the discarded files. This can be natively done on Linux systems.

Downloading & extracting in parallel, maximizing performance?

1 Answers