Url works in normal browser but gives timeout in python requests or urllib or webdriver

user_11836214 0
Viewed 126

I have looked up multiple solutions but none works. If you open the URL in a normal browser, it will return data. I am unsure what is happening that it keeps loading if a timeout is not given when the python request is sent.

import urllib.request, json 
with urllib.request.urlopen("https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY") as url:
    data = json.loads(url.read().decode())
    print(data)

I have tried a solution from here as well API request works in browser, but, doesn't work in postman or python requests module

import requests
url = "https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY"

payload = {}
headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-IE,en-GB;q=0.9,en-US;q=0.8,en;q=0.7',
    }

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)
python-requests web-scraping selenium urllib python
1 Answers

Try to set User-Agent header:

import json
import requests

url = "https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0"
}

data = requests.get(url, headers=headers).json()
print(json.dumps(data, indent=4))

Prints:

{
    "records": {
        "expiryDates": [
            "16-Jun-2022",
            "23-Jun-2022",
            "30-Jun-2022",
            "07-Jul-2022",
            "14-Jul-2022",
            "28-Jul-2022",
            "25-Aug-2022",
            "29-Sep-2022",
            "29-Dec-2022",
            "30-Mar-2023"
        ],
        "data": [
            {
                "strikePrice": 25500,
                "expiryDate": "30-Jun-2022",
                "PE": {
                    "strikePrice": 25500,
                    "expiryDate": "30-Jun-2022",
                    "underlying": "BANKNIFTY",
                    "identifier": "OPTIDXBANKNIFTY30-06-2022PE25500.00",

...and so on.
user_100359850
Related