Modify `pd.read_html()` when reading from a website that requires you to hit "Accept" for Cookies - HTTPError: HTTP Error 500: Internal Server Error?

user_6366770 0
Viewed 210

I have always been able to run this line of code without issues to return the table from the Bank of England as a dataframe:

import pandas as pd
pd.read_html('https://www.bankofengland.co.uk/boeapps/database/Rates.asp?Travel=NIxIRx&into=GBP')[0]

enter image description here

However, it now returns the error:

HTTPError: HTTP Error 500: Internal Server Error

I'm pretty sure this is because of newly implemented cookie requirements on the website (see image). Is there still a way to access this with read_html(). I couldn't find any questions related to read_html() and cookies, but this thread might help solve ( Cannot catch HTTP Error 500: Internal Server Error). I was hoping for a simple solution with the read_html() library, but I might have to use requests / beautifulsoup.

Can anyone help me return the table from that link as output? It might not be possible now with one line of code of read_html():

python-requests pandas web-scraping beautifulsoup python
1 Answers

You need to inject User-Agent

import requests
import pandas as pd

headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36'}

url = "https://www.bankofengland.co.uk/boeapps/database/Rates.asp?Travel=NIxIRx&into=GBP"

req = requests.get(url,headers=headers)

table = pd.read_html(req.text, attrs = {"class":"table"} )

df = table[0]#.to_csv('score.csv',index = False)

print(df)

Output:

            Currency  £1 13 Oct 2021  £1 52 wk-high  £1 52 wk-low
0    Australian Dollar          1.8493         1.9103        1.7434
1      Canadian Dollar          1.6970         1.7769        1.6880
2         Chinese Yuan          8.7618         9.1286        8.5980
3         Czech Koruna         29.8762        30.7526       28.6068
4         Danish Krone          8.7644         8.7869        8.1111
5                 Euro          1.1779         1.1815        1.0902
6     Hong Kong Dollar         10.6069        11.0357       10.0020
7     Hungarian Forint        424.4322       430.6888      386.7975
8         Indian Rupee        102.7158       105.1990       94.8636
9       Israeli Shekel          4.4054         4.6524        4.2677
10        Japanese Yen        154.5301       156.2211      134.9730
11   Malaysian ringgit          5.6713         5.9233        5.3545
12  New Zealand Dollar          1.9596         2.0030        1.8646
13     Norwegian Krone         11.6286        12.3972       11.4222
14        Polish Zloty          5.3925         5.4644        4.8446
15       Russian Ruble         98.2299       107.0946       96.5495
16         Saudi Riyal          5.1133         5.3296        4.8394
17    Singapore Dollar          1.8435         1.8907        1.7564
18  South African Rand         20.2195        21.5353       19.1216
19    South Korean Won       1621.0728      1630.4056     1442.6832
20       Swedish Krona         11.8859        12.0659       11.0226
21         Swiss Franc          1.2627         1.3035        1.1753
22       Taiwan Dollar         38.2719        39.7162       36.8693
23           Thai Baht         45.2616        46.3459       39.6758
24        Turkish Lira         12.3676        12.3676        9.6744
25           US Dollar          1.3633         1.4211        1.2904
user_128484110
Related