I am trying to get a Python script that uses the requests module to request data through a proxy. According to the requests documentation:
You can also configure proxies by setting the environment variables HTTP_PROXY and HTTPS_PROXY.
However, as far as I can tell, this only works if your target URL references a hostname, and fails when the target URL references an IP address. For example, I am trying to fetch a resource http://overcloud-public-vip:5000. The hostname (overcloud-public-vip) is not resolvable locally, but can be resolved by the proxy to 192.168.30.6. This works:
>>> import os
>>> import requests
>>> os.environ['HTTP_PROXY'] = 'http://localhost:8123'
>>> requests.get('http://overcloud-public-vip:5000')
<Response [300]>
But if I try exactly the same thing, using an ip address rather than a hostname, it fails:
>>> import os
>>> import requests
>>> os.environ['HTTP_PROXY'] = 'http://localhost:8123'
>>> requests.get('http://192.168.30.6:5000')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/my/project/env/lib/python2.7/site-packages/requests/api.py", line 67, in get
return request('get', url, params=params, **kwargs)
File "/my/project/env/lib/python2.7/site-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/my/project/env/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/my/project/env/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/my/project/env/lib/python2.7/site-packages/requests/adapters.py", line 437, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.30.6', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f89e7d52950>: Failed to establish a new connection: [Errno 110] Connection timed out',))
I can confirm (using strace -e trace=connect python) that the module
never makes any attempt to contact the proxy server in this second
case.
This is particularly frustrating because even if I pass in a URL using a hostname, the code will ultimately retrieve a service catalog from the remote server that itself includes ip addresses, so I have simply postponed the failure by one step.
This behavior seems at odds with the documentation. I'm using requests version 2.9.1:
>>> requests.__version__
'2.9.1'
Is this a bug in requests, or a bug in my brain?