Python - How to handle HTTPS request with (Urllib2 + SSL) though a HTTP proxy

Viewed 7125

I am trying to test a proxy connection by using urllib2.ProxyHandler. However, there probably some situation that I am going to request a HTTPS website (eg: https://www.whatismyip.com/)

Urllib2.urlopen() will throw ERROR if request a HTTPS site. So I tried to use a helper function to rewrite the URLOPEN method.

Here is the helper function:

def urlopen(url, timeout):
    if hasattr(ssl, 'SSLContext'):
        SslContext = ssl.create_default_context()
        SslContext.check_hostname = False
        SslContext.verify_mode = ssl.CERT_NONE
        return urllib2.urlopen(url, timeout=timeout, context=SslContext)
    else:
        return urllib2.urlopen(url, timeout=timeout)

This helper function based on answer

Then I use:

urllib2.install_opener(
     urllib2.build_opener(
         urllib2.ProxyHandler({'http': '127.0.0.1:8080'})
     )
)

to setup http proxy for urllib.opener.

Ideally, it should working when i request a website by using urlopen('http://whatismyip.com', 30) and it should pass all traffic through http proxy.

However, the urlopen() will fall into if hasattr(ssl, 'SSLContext') all the time even if it is a HTTP site. In addition, HTTPS site is not using HTTP proxy either. This cause the HTTP proxy become invalid and all traffic going through unproxied network

I also tried this answer to change HTTP into HTTPS urllib2.ProxyHandler({'https': '127.0.0.1:8080'}) but it still not working.

My proxy is working. If i am using urllib2.urlopen() instead of the rewrite version urlopen(), it works for HTTP site.

But, I do need consider the suitation if the urlopen gonna need to be used on a HTTPS ONLY site.

How to do that?

Thanks

UPDATE1: I cannot get this work with Python 2.7.11 and some of server working properly with Python 2.7.5. I assue it is python version issue.

Urllib2 will not go through HTTPS Proxy so all HTTPS web address will failed to use proxy.

3 Answers
Related