Python Requests - POST timeout specification with large files

Viewed 261

Background

I have a python app, built using requests, that is used to upload files from client sites to a web server, using POST.

These files are usually small (1-300 KB), but sometime larger (15-20MB). Usually the uploads take a few seconds, however for large files over slow networks may take minutes to complete.

Problem

I'm having a problem figuring out how to use requests timeout in a rational way to handle sending large uploads using POST over slow networks (where the POST may take 1-2 min to complete).

What I'd like

I'd like to be able to declare a session and than a POST using the session, so that

a) an initial timeout was small (so network/gateway/... connection problems etc get detected quickly), BUT

b) a subsequent timeout that is long, so that after the connection is established, but the data takes a few minutes to upload, it won't timeout.

I can't seem to figure out how to do that

I'm also a bit confused by how/what/where the timeout parameters is used when specified as a tuple in conjuction with POST (looks like I'm not alone: https://stackoverflow.com/a/63994047/9423009)

Specifically to illustrate this (meta code - my production code is below), if I have a file to POST that may take 1-2 minute to upload:

file_to_upload = '/path_to_a_big_file'

my_session.post(
    timeout=2,
    files=file_to_upload
)
# above will timeout if POST takes > 2 seconds

my_session.post(
    timeout=60,
    files=file_to_upload
)
# above will succeed if POST takes 40 seconds, BUT will also take 60 seconds
# to throw any exceptions of problems with any routine type network/gateway 40X 
# type problems

my_session.post(
    timeout=(2, 60),
    files=file_to_upload
)
# THIS WILL ALSO TIMEOUT AFTER 2 SECONDS!?

So based on above, how do you specify a small initial 'make connection' timeout, and then a longer, separate, timeout for a POST to complete sending?

Actual code and Additional Stuff

As the sending sites may have variable speed networks, and to handle flaky network problems etc, I use urllib3's Retry to generate Sessionss (courtesy of some great code at https://www.peterbe.com/plog/best-practice-with-retries-with-requests).

With this code, I have a small'ish initial timeout, that the Retry code will increase for a certain amount of times until things fail. But I don't believe this affects the problem here:

def requests_retry_session(
    retries=3,
    backoff_factor=0.3,
    status_forcelist=(500, 502, 504),
    session=None,
) -> requests.Session:
    """ Return requests session using Retry to automatically retry on failures."""

    # add POST to list of methods to retry on
    methods = frozenset({'DELETE', 'GET', 'HEAD', 'OPTIONS', 'PUT', 'POST', 'TRACE'})

    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        status=retries,
        backoff_factor=backoff_factor,
        method_whitelist=methods,
        status_forcelist=status_forcelist,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)

    return session

# ...

        # send file 

        with open(file_to_send, 'rb') as fh:

            file_arg = [(server_key, fh)]
            with requests_retry_session() as s:

                # try to specify small initial, and long subsequent POST timeout
                # but doesn't work - if POST takes > 2 seconds it still will
                # timeout
                timeout=(2.0, 60.0)
                
                response = s.post(
                    url,
                    headers=headers,
                    data={},
                    files=file_arg,
                    timeout=timeout,
                    proxies=proxies
                )
                response.raise_for_status()
0 Answers
Related