Python "requests" module does not respect http_proxy environment variable when target URL uses an ip address?

Viewed 2040

I am trying to get a Python script that uses the requests module to request data through a proxy. According to the requests documentation:

You can also configure proxies by setting the environment variables HTTP_PROXY and HTTPS_PROXY.

However, as far as I can tell, this only works if your target URL references a hostname, and fails when the target URL references an IP address. For example, I am trying to fetch a resource http://overcloud-public-vip:5000. The hostname (overcloud-public-vip) is not resolvable locally, but can be resolved by the proxy to 192.168.30.6. This works:

>>> import os
>>> import requests
>>> os.environ['HTTP_PROXY'] = 'http://localhost:8123'
>>> requests.get('http://overcloud-public-vip:5000')
<Response [300]>

But if I try exactly the same thing, using an ip address rather than a hostname, it fails:

>>> import os
>>> import requests
>>> os.environ['HTTP_PROXY'] = 'http://localhost:8123'
>>> requests.get('http://192.168.30.6:5000')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/my/project/env/lib/python2.7/site-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/my/project/env/lib/python2.7/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/my/project/env/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/my/project/env/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/my/project/env/lib/python2.7/site-packages/requests/adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.30.6', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f89e7d52950>: Failed to establish a new connection: [Errno 110] Connection timed out',))

I can confirm (using strace -e trace=connect python) that the module never makes any attempt to contact the proxy server in this second case.

This is particularly frustrating because even if I pass in a URL using a hostname, the code will ultimately retrieve a service catalog from the remote server that itself includes ip addresses, so I have simply postponed the failure by one step.

This behavior seems at odds with the documentation. I'm using requests version 2.9.1:

>>> requests.__version__
'2.9.1'

Is this a bug in requests, or a bug in my brain?

0 Answers
Related