Why "requests" module timeout doesn't work in python3?

user_3580433 0
Viewed 556

I want to fetch content of a URL and in cases I face 50X HTTP error codes, retry for 10 times with 0.5 seconds delay between retries. I also want to have a 1 second timeout for my requests. To aim this goal I tried the following program:

#!/usr/bin/python3

import sys
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter    

URL = "http://<DOMAIN>/Loader.aspx?ParTree=15131H&i=" # + stockid


def get_data(stockid):

    print("1")
    req_session = requests.Session()

    print("2")
    retries = Retry(total = 10, 
                    backoff_factor = 0.5,
                    status_forcelist = [500, 502, 503, 504])

    print("3")
    req_session.mount("http://", HTTPAdapter(max_retries=retries))

    print("4")
    page_content = req_session.get(URL + stockid, timeout=1).content

    print("5")

get_data(sys.argv[1])

Unfortunately it freezes for minutes after printing 4 on the screen. What is wrong?

urllib3 python-requests python-3.x
1 Answers

I'm a Windows User, first of all. Your timeout works fine. Here the backoff_factor creates the issue. Try:

#!/usr/bin/python3

import sys
import requests
from requests.packages.urllib3.util.retry import Retry # requests.packages. is not necessary
from requests.adapters import HTTPAdapter    

URL = "http://<DOMAIN>/Loader.aspx?ParTree=15131H&i=" # + stockid


def get_data(stockid):

    print("1")
    req_session = requests.Session()

    print("2")
    retries = Retry(total = 10, 
                    backoff_factor = 0.5,
                    status_forcelist = [500, 502, 503, 504])
    requests.packages.urllib3.util.Retry.BACKOFF_MAX = 0.5 # required
    # requests.packages. is not necessary
    # it isin't necessary you have to mount this way
    print("3")
    adapter = HTTPAdapter(max_retries=retries)
    req_session.mount('http://', adapter)
    req_session.mount('https://', adapter)

    print("4")
    page_content = req_session.get(URL + stockid, timeout=1).content

    print("5")

get_data(sys.argv[1])

This way is also possible:

retries = RetryRequest(
        total=10,
        backoff_factor=0.5,
        status_forcelist=[500, 502, 503, 504],
        max_backoff=backoff_factor)

It is documented that, when you use the backoff_factor:

It will never be longer than Retry.BACKOFF_MAX ( which is by default 120).

So when you try your link, the BACKOFF_MAX preceeds your backoff_factor, which increase your execution time. Therefore, setting:

requests.packages.urllib3.util.Retry.BACKOFF_MAX = 0.5

will help you.
But this error shouldn't be raised since it defined as:

min(self.BACKOFF_MAX, backoff_value)

in:

urllib3.util.retry
user_130628130
Related