Proper pattern for Python `requests` result checking?

user_479426 0
Viewed 41

I find my code scattered with this pattern:

response = requests.get(url, headers=HEADERS)
if response.status_code == 429:
    # sleep and try again until some limit
elif response.status_code != 200:
    response.raise_for_status()
result=response.json()

Of course, I can make my own wrapper function, but ... this seems like a really common thing. Is there an existing slightly-higher-level call that already exists that I should be using?

python-requests python
1 Answers

Requests provides access to urllib3's Retry class which allows you to create a session which will retry connections based on a variety of situations.

The following demonstrates retrying when getting 429 status back:

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

##### optional logging - only needed here to show retries is working #####
import logging
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
req_log = logging.getLogger('requests.packages.urllib3')
req_log.setLevel(logging.DEBUG)
req_log.propagate = True
##### end logging #####

s = requests.session()
# total defaults to 10 - omit backoff_factor to retry as fast as possible
retries = Retry(total=5, backoff_factor=2, status_forcelist=[ 429 ])
# Associate this session with urls starting with https://
s.mount('https://', HTTPAdapter(max_retries=retries))

# See https://httpbin.org/#/Status_codes for details (takes a comma-separated list of codes to return at random)
response = s.get("https://httpbin.org/status/429")
print(response)
user_92031100
Related