ReadTimeout on a url in requests get?

user_6387095 0
Viewed 88
url="https://www.nseindia.com/companies-listing/corporate-filings-actions"
# second url I am trying 
# url = "https://www.nseindia.com/json/CorporateFiling/CF-corpactions-equity.json"

I am trying to get the json response.

I have tried a bunch of the suggested actions in similar posts.

import requests as r
headers = {
        "Connection": "keep-alive",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
    }
resp = r.get(url, headers=headers)
resp.json()
## Stuck for 20 minutes no response.

I tried a timeout:

resp = r.get(url, headers=headers, timeout=20)
# Timed out

Also tried a catch except block but still not working.

What am I doing wrong here?

python-requests python
2 Answers

On UNIX, I use the following Timeout class that makes use of UNIX SIGALARM to interrupt the execution of code in such cases:

import signal

class TimeoutError(Exception):
    """
    Custom error for Timeout class.
    """

    pass


class Timeout:
    """
    A timeout handler with context manager.
    Based on UNIX signals.
    """

    def __init__(self, seconds=1, error_message="Timeout"):
        self.seconds = seconds
        self.error_message = error_message

    def handle_timeout(self, signum, frame):
        raise TimeoutError(self.error_message)

    def __enter__(self):
        signal.signal(signal.SIGALRM, self.handle_timeout)
        signal.alarm(self.seconds)

    def __exit__(self, type, value, traceback):
        signal.alarm(0)

You can use it very simply like this:

with Timeout(20):
    try:
        resp = r.get(url, headers=headers)
        resp.json()
    except TimeoutError:
        print('Timed out')
print('done')
user_31900760

Figured it out:

resp = r.get(url, stream=True, timeout=20, headers=header)

I am still looking for why this works and the earlier ones didn't.

user_63870950
Related