How to get full server response message in Requests (or urllib3.response)?

user_10254804 0
Viewed 4400

I encountered problems with Google Sheet API these few days, all response message I can get from Requests (by way of requests_oauthlib extension) is the headers & 400 Bad Request. Requests version is 2.19.1. This happened with several endpoints (v4), like spreadsheets.values.append, spreadsheets.values.batchUpdate, spreadsheets.values.update, etc.

I've wasted hours making sure I have formed the correct data structure to no avail, and only then I realized I didn't get the whole response message from the API server.

It seems since requests.Response object that gets returned from r = request.Request('https://httpbin.org') is from a requests.HTTPAdapter object, which in turn is sort of a wrapper for urllib3.response.HTTPResponse. So I tried to use r.reason, r.raw._original_response, r.raw._body, r.raw.msg which didn't give me anything but object reference & None (or just empty string).

Long story short, I gave up for a while & when got back to it I realized I should've tried using Google's API explorer that i've been ignoring, and it turned out there is an error message in the response, and the 400 error wasn't due to any bad payload/data structure form.

{   "error": {
    "code": 400,
    "message": "Invalid data[0]: Requested writing within range [mysheet!A3:AH3], but tried writing to row [4]",
    "status": "INVALID_ARGUMENT"   } 
}

My question is:

Was I wrong in how I debug connection response from requests & actually there's a method to get the full raw message from the server in requests like a full debug mode in curl? Or is this message is actually only sent if the API request comes from Google API explorer?

urllib3 python-requests httpresponse
2 Answers

The error message you're looking for is located in the response body. This can be accessed with response.text (or response.json(), as the particular body is JSON). If, as your self-answer suggests, you want to get the body from an exception raised by raise_for_status(), you can just go through the exception's response attribute: exception.response.text or exception.response.json().

user_7441780

I found a closed issue in requests repo from last year titled as "Suggestion: Improve the raise for status to include response body if there is one" issued by westover, in which he wrote:

Looking for more details on the HTTPError exception. The API I am working with sends back a 400 but also has a text/json body in it. It would be nice if the raise_for_status() function included the response text if there was any.

He also provided a little workaround function that became the base of the reason that discussion diverted into a pull request discussion:

import requests
from requests.exceptions import HTTPError
def expanded_raise_for_status(res):
    """
    Take a "requests" response object and expand the raise_for_status method to return more helpful errors
    @param res:
    @return: None
    """
    try:
        res.raise_for_status()
    except HTTPError as e:
        if res.text:
            raise HTTPError('{} Error Message: {}'.format(str(e.message), res.text))
        else:
           raise e
    return

I adopted this to my existing class & essentially as jwodder also pointed out, it extends/hijacks Response.raise_for_status() method that calls HTTPError(RequestsException) class whenever raise_for_status() is used by users to serve the error message from server response, usually in the form of assert response.status_code == 200, raise_for_status(). This is achieved by adding Response.text (or content / json methods) to the exception message and returns the response body, instead of current code return <self.status_code, self.reason, self.url> which translates to 400 Bad Requests <url>.

def raise_for_status(self):
    """Raises stored :class:`HTTPError`, if one occurred."""

    http_error_msg = ''
    if isinstance(self.reason, bytes):
        # We attempt to decode utf-8 first because some servers
        # choose to localize their reason strings. If the string
        # isn't utf-8, we fall back to iso-8859-1 for all other
        # encodings. (See PR #3538)
        try:
            reason = self.reason.decode('utf-8')
        except UnicodeDecodeError:
            reason = self.reason.decode('iso-8859-1')
    else:
        reason = self.reason

    if 400 <= self.status_code < 500:
        http_error_msg = u'%s Client Error: %s for url: %s' % (self.status_code, reason, self.url)

    elif 500 <= self.status_code < 600:
        http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)

    if http_error_msg:
        raise HTTPError(http_error_msg, response=self)

(In this reply he further laid out the reason why he thinks it's worthy of a feature, but the pull request got rejected and I think the maintainer also had good points), this might leads to other exceptions or more complications if not used thoughtfully/sparingly given the popularity of the module and the broad reasons of its usage.

user_102548040
Related