urllib.openurl() doesn't open URL

Gal Eindor 0
Viewed 18

I'm trying to use urllib.openurl() like that:

import urllib
html = urllib.request.urlopen(url="https://www.otcmarkets.com")

But it just doesn't work for some reason. If I give different HTTPS url like 'https://www.google.com' it works but for some websites like this one I just can't open the url. Can I do something to make it work? Do you have another way to extract html from website?

urlopen urllib url python-3.x python
1 Answers

The website checks if you set a User-Agent in the headers. The requests-package makes it easy to use urllib.
You can do what you want by using something like that:

import requests
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'}
resp = requests.get("https://www.otcmarkets.com", headers=headers)
resp.text
bitflip0
Related