Is it possible to recover the HTTP response from a RetryError? (Python requests library)

user_388916 0
Viewed 201

I am dealing with a HTTP API that, unfortunately, has some stability issues for the time being. It will now and then throw a 500 server error when load gets too high, and it starts tripping over itself.

To make this less of a problem, I've enabled the retry mechanism for 500 server errors:

def _start_http_session() -> requests.Session:
    """Starts a HTTP session"""

    adapter = requests.adapters.HTTPAdapter(max_retries=RETRY_STRATEGY)
    session = requests.Session()
    session.mount("https://", adapter)
    session.mount("http://", adapter)

    return session

This works great, but the problem is that I can no longer find a reference to the HTTP response in case of an error. I have a top level error handler that looks something like this:

if (
    isinstance(error, requests.RequestException)
    and error.response is not None
):
    req = error.request
    resp = error.response

    # Log request and response ...

This used to work great when I did response.raise_for_status(), but it no longer works because RetryError doesn't include a reference to the response. Instead, the error now gets logged as a generic exception, with no information about the response from the server:

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8088): Max retries exceeded with url: /test (Caused by ResponseError('too many 500 error responses'))

I really want to be able to log the server response, so I can provide more information when filing bug reports towards the API.

Is there any way to recover the HTTP response in case of errors when using a retry strategy?

python-requests python
1 Answers

As far as I know is there no built-in way to capture the different responses while using this retry strategy.

It is possible though with a bit of ugly decorating code of urllib3, where you force your logging into the retry logic. Here I decorated the increment method of Retry, which is called after each request.

import requests
import requests.adapters
from urllib3.util.retry import Retry


def increment_decorator(inc):
    def wrapper(*args, **kwargs):

        # Peform logging here on HTTPResponse instance
        response = kwargs["response"]
        print(response.read(decode_content=True))
        print(response.status)
        print(response.headers)

        value = inc(*args, **kwargs)
        return value
    return wrapper


# Decorate imported increment
Retry.increment = increment_decorator(Retry.increment)
RETRY_STRATEGY = Retry(total=2, status_forcelist=[200])


def _start_http_session() -> requests.Session:
    """Starts a HTTP session"""

    adapter = requests.adapters.HTTPAdapter(max_retries=RETRY_STRATEGY)
    session = requests.Session()
    session.mount("https://", adapter)
    session.mount("http://", adapter)

    return session


with _start_http_session() as session:
    try:
        response = session.get("https://httpbin.org/uuid")
    except Exception as error:
        print(error)
user_126640980
Related