The requests documentation (link) mentioned that a session is what allows some parameters to persist across requests. My use case is simple; because I sit behind a corporate proxy and firewall, I need to set the proxy parameters proxies (as mentioned in the title) in a session and I don't want to have to set it for every request.
Supposedly, you can do the following (directly copied from the proxies section):
import requests
proxies = {
'http': 'http://10.10.1.10:3128',
'https': 'http://10.10.1.10:1080',
}
session = requests.Session()
session.proxies.update(proxies)
session.get('http://example.org')
This should allow you to set proxies, without stating them in the request itself. Thus my session function looks like this below:
def requests_setup():
# setup proxy
proxies = {'http': f'http://someproxy:8080',
'https': f'http://someproxy:8080'}
# initialize session
session = requests.Session()
# Part 1: set up proxy
session.proxies.update(proxies)
# Part 2: add certificate
session.verify = r'SOME_CERT_BUNDLE.pem'
return session
Get request example that results in an error
# making an example get request
setup = requests_setup()
url = "https://example.com"
r = setup.get(f"{url}", timeout=5)
Posting the full traceback below, but the following errors seems to be the problem. And my understanding of this is that the ssl cert verification did not go through for some reason (as suggested by the trace, I believe it is because proxy was not included; for a session without the verify parameter set, it would instead result in a sslCertVerification error during the request that worked below).
Error 1 ...
socket.timeout: _ssl.c:1074: The handshake operation timed out
... leading to Error 2
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1074: The handshake operation timed out')))
... and finally Error 3
requests.exceptions.ProxyError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1074: The handshake operation timed out')))
The silver lining is that this was solved, eventually, by specifying the parameter in the request body.
setup = utils.requests_setup()
# making an example get request
url = "https://example.com"
proxies = {'http': f'http://someproxy:8080',
'https': f'http://someproxy:8080'}
r = setup.get(f"{url}", timeout=5, proxies=proxies)
But why is that the case? I can see clearly that my session's proxy attributes are initialized . But for some reason it was not utilized in the get request made using that session.
PS: There might be questions about why my proxy is prefixed with http for both cases. It is purely because we don't have a standalone https proxy server. The request also fails when I use a "HTTPS" prefix instead there.
PPS: example.com is not the site used. I have tried to use google.com, or others (such as the API I am trying to call), but that did not change the results.
Actual Error Traceback
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connection.py", line 359, in connect
conn = self._connect_tls_proxy(hostname, conn)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connection.py", line 506, in _connect_tls_proxy
ssl_context=ssl_context,
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\util\ssl_.py", line 450, in ssl_wrap_socket
sock, context, tls_in_tls, server_hostname=server_hostname
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\ssl.py", line 423, in wrap_socket
session=session
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\ssl.py", line 870, in _create
self.do_handshake()
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
socket.timeout: _ssl.c:1074: The handshake operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\util\retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1074: The handshake operation timed out')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1074: The handshake operation timed out')))
Information for reproducing the issue:
OS version: 'Windows-10-10.0.18362-SP0'
Python version: '3.7.11 (default, Jul 27 2021, 09:42:29) [MSC v.1916 64 bit (AMD64)]'
Requests version: '2.26.0'