Proxy with urllib2

user_106697 0
Viewed 128828

I open urls with:

site = urllib2.urlopen('http://google.com')

And what I want to do is connect the same way with a proxy I got somewhere telling me:

site = urllib2.urlopen('http://google.com', proxies={'http':'127.0.0.1'})

but that didn't work either.

I know urllib2 has something like a proxy handler, but I can't recall that function.

urllib2 proxy python
7 Answers

You can set proxies using environment variables.

import os
os.environ['http_proxy'] = '127.0.0.1'
os.environ['https_proxy'] = '127.0.0.1'

urllib2 will add proxy handlers automatically this way. You need to set proxies for different protocols separately otherwise they will fail (in terms of not going through proxy), see below.

For example:

proxy = urllib2.ProxyHandler({'http': '127.0.0.1'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com')
# next line will fail (will not go through the proxy) (https)
urllib2.urlopen('https://www.google.com')

Instead 

proxy = urllib2.ProxyHandler({
    'http': '127.0.0.1',
    'https': '127.0.0.1'
})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
# this way both http and https requests go through the proxy
urllib2.urlopen('http://www.google.com')
urllib2.urlopen('https://www.google.com')
user_11188190

To use the default system proxies (e.g. from the http_support environment variable), the following works for the current request (without installing it into urllib2 globally):

url = 'http://www.example.com/'
proxy = urllib2.ProxyHandler()
opener = urllib2.build_opener(proxy)
in_ = opener.open(url)
in_.read()
blueyed_156900
Related