The following code fetches only a 404 error page from oddsportal.com site:
import requests
#url = r'https://www.reuters.com/world/europe/zelenskiy-alleges-torture-war-crimes-evidence-recaptured-northeast-2022-09-16/'
#url = r'https://www.oddsportal.com/soccer/japan/j1-league/iwata-cerezo-osaka-MmdhGr2D/'
#url = r'https://www.oddsportal.com/soccer/germany/bundesliga/vfb-stuttgart-eintracht-frankfurt-MVMaYXD4/'
url = r'https://www.oddsportal.com/soccer/italy/serie-a/empoli-as-roma-CrUS7raC/'
page = requests.get(url, allow_redirects=True)
print (page.content)
#open('Reuters.html', 'wb').write(page.content)
open('Empoli-Roma.html', 'wb').write(page.content)
However, if you copy any of the their URLs (some are commented) and paste them into a browser, then they work.
The script works correctly on other sites that I have tried (I have included a piece of news from Reuters in a comment, which works as expected).
Please let me know how I can open those URLs in Python.
Unfortunately, I can’t use Selenium, because it is too slow (I need to get quite a lot of odds pages and subpages) or it needs to be done in a background. So I plan to decipher Oddsportal’s JavaScript, but at first I need to open a necessary page.
Thank you!