urllib.request SSL Connection Python 3

user_7468817 0
Viewed 5228

I'm trying to parse the data from this url:

https://www.chemeo.com/search?q=show%3Ahfus+tf%3A275%3B283

But I think this is failing because the website uses SSL TLS 1.3. How can I enable my Python script, below, to connect using SSL in urllib.request?

I've tried using an SSL context but this doesn't seem to work.

This is the Python 3.6 code I have:

import urllib.request
import ssl
from bs4 import BeautifulSoup

scontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
chemeo_search_url = "https://www.chemeo.com/search?q=show%3Ahfus+tf%3A275%3B283"

print(chemeo_search_url)

with urllib.request.urlopen(chemeo_search_url, context=scontext) as f:
    print(f.read(200))
web-scraping urllib2 python-3.x ssl python
1 Answers

Try:

ssl.PROTOCOL_TLS

From the docs on "PROTOCOL_SSLv23":

Deprecated since version 2.7.13: Use PROTOCOL_TLS instead.

note:

Be sure to have the CA certificate bundles installed, like on a minimal build of alpine linux, busybox, the certs have to be installed. Also sometimes if python wasn't compiled with SSL support, it might be necessary to to do so. Also depending on which version of OpenSSL has been compiled will determine which features for SSL will be usable.

Also note chemeo site doesn't use TLSv1.3 ... it is still experimental and not all that secure at the time of this writing, they currently support tls 1.0, 1.1, 1.2 using "letsencrypt" as their cert provider.

enter image description here

user_89930120
Related