Python requests - quickly know if response is json parsable

Question

I wrote a certain API wrapper using Python's requests library.

When it gets a response using requests.get, it attempts to parse as json and takes the raw content if it doesn't work:

resp = requests.get(url, ...)  

try:
    resp_content = resp.json()
except ValueError:
    resp_content = resp.content

return resp_content

This is correct for my purposes. The problem is how long it takes when the downloaded response is an image file, for example, if it is large, then it takes an extremely long time between entering the try, and failing the json parse and entering the except.

(I don't know if it takes super long for the .json() to error at all, or if once it errors it then takes a while to get into the except.)

Is there a way to see if resp is json-parsable without attempting to parse it with .json()? Something like resp.is_json, so I can instantly know which branch to take (resp.json() or resp.content), instead of waiting 30 seconds (large files can take minutes).

EDIT:

As noted, this slowness is not typical for requests json parsing. It may have to do with the nature of the data I am receiving (it is from Salesforce REST API, retrieving Attachment object's 'Body' field).

Even though it's a workaround, I will put my solution here in case the strategy helps anyone else. I realized where I'm making the call, I usually know if I expect the response to be binary data, so I can pass a keyword argument to my wrapper function that tells it to skip the json parsing attempt.

def SalesforceWrapper(..., attempt_json=True):

    resp = requests.get(url, ...)  

    try:
        if attempt_json:
            resp_content = resp.json()
        else:
            resp_content = resp.content
    except ValueError:
        resp_content = resp.content

    return resp_content

and then I pass attempt_json=False when I expect the response to be a file's data, not a lil JSON response.

user_11641690 · Answer

You can check if the content-type application/json is in the response headers:

'application/json' in response.headers.get('Content-Type')

user_1033581 · Answer

(Addressing Daniel Kats comment in a previous response)

You could check if the returned headers include Content-Type application/json:

response.headers.get('Content-Type').startswith('application/json')

By using startswith, you're accounting for all allowed formats from https://www.w3.org/Protocols/rfc1341/4_Content-Type.html.

That doesn't guaranty that it will be a valid JSON, but at least that will catch responses which aren't being declared as JSON.

user_5795878 · Answer

If using Session and not directly requests.(METHOD)

from requests import Session
from simplejson.errors import JSONDecodeError

class MySession(Session):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)

    def request(self, *args, **kwargs):
        res = super().request(*args, **kwargs)
        json = res.json
        def wrapper():
            try:
                return json()
            except JSONDecodeError:
                return None
        res.json = wrapper
        return res
        
session = MySession()
res = session.get("https://api64.ipify.org")
if res.json():
    print("ok")

Python requests - quickly know if response is json parsable

5 Answers