Streaming upload of a slice of a (large) file using python-requests

Viewed 283

I need to do a streaming upload (i.e., not load the full file part in memory) of a slice of a large (multi-GB) file, using python-requests.

I've looked around in the doc and on Stack Overflow, and haven't found a working way to do it (again, without loading the full slice in memory).

Here's the code I have:

class FileSlice(AbstractContextManager):
    """
    File-like object that only reads a slice of a file

    Inspired by stackoverflow.com/a/29838711/593036, but actually works.
    """

    def __init__(self, filepath: str, seek_from: int, read_limit: int):
        self.filepath = filepath
        self.seek_from = seek_from
        self.read_limit = read_limit
        self.n_seen = 0

    def __enter__(self):
        self.f = open(self.filepath, "rb")
        self.f.seek(self.seek_from)
        return self

    def __len__(self):
        total_length = os.fstat(self.f.fileno()).st_size
        return min(self.read_limit, total_length - self.seek_from)

    def read(self, n=-1):
        if self.n_seen >= self.read_limit:
            return b""
        remaining_amount = self.read_limit - self.n_seen
        n_to_read = remaining_amount if n < 0 else min(n, remaining_amount)
        self.n_seen += n_to_read
        return self.f.read(n_to_read)

    def __iter__(self):
        yield self.read(n=io.DEFAULT_BUFFER_SIZE)

    def __exit__(self, *args):
        self.f.close()

Then the actual request:

with FileSlice(filepath, seek_from=i * chunk_size, read_limit=chunk_size) as data:
    r = requests.put(presigned_url, data=data)
    r.raise_for_status()

This seems pretty complex, so I'm wondering:

  • if I'm missing a simpler way
  • if my approach is correct.

Thank you!

0 Answers
Related