How to retry 104 connection reset by peer connection errors in python requests library without try except

user_5595258 0
Viewed 1379

I have the following Session object :

    import requests
    from requests.adapters import Retry, HTTPAdapter
    from requests.sessions import Session

    retry_strategy: Retry = Retry(
        total=5,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["HEAD", "GET", "OPTIONS", "POST"],
        backoff_factor=1
    )
    http_adapter = HTTPAdapter(max_retries=retry_strategy)
    session = requests.Session()
    session.mount("https://", http_adapter)

and once in a while the server seems to break the connection with an RST TCP-packet resulting in the following Error:

{"isError": true,
 "type": "ConnectionError",
 "message": "[Errno 104] Connection reset by peer",
 "traceback": "...  omitted for brevity ...
    File "/var/task/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)    
    File "/var/task/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)   
    File "/var/task/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs) 
    File "/var/task/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)"
  1. Is it possible to inject a Retry object in a Session object so that this particular ConnectionError is automatically retried?
  2. Is there any other way to retry automatically without any additional custom logic such as wrapping it in a try... except statement or creating a custom HTTPAdapter?
  3. I have seen multiple posts regarding 104 connection reset by peer errors, but no one asking how to retry it automatically. Why is that? I assume I won't be the only one with this problem?
python-requests connection-reset python-3.x
3 Answers

Give a look at https://github.com/jd/tenacity

I usually do something like this

from tenacity import wait_exponential

@retry(wait=wait_exponential(multiplier=1, min=4, max=10))
    def wait_exponential_1():
        # my request here

You can create a function that yields a session wrapped with tenacity.

user_46234190

I have eventually ended up with backoff:

@backoff.on_exception(
    wait_gen=backoff.expo,
    exception=requests.ConnectionError,
    max_time=time_units.in_s.SEC_15,
)
def robust_session_request(*args, **kwargs) -> requests.Response:
    return session.request(*args, **kwargs)
user_74832390

You would need to provide the "connect" argument:

  from requests import adapters
  import urllib3

  sync_session = requests.Session()
  retry_adapter = adapters.HTTPAdapter(max_retries=urllib3.Retry(total=1, connect=1))
  session.mount('https://', retry_adapter)
  session.mount('http://', retry_adapter)

Test:

class AdapterTest(unittest.TestCase):

  @mock.patch.object(urllib3.connectionpool.HTTPConnectionPool, '_make_request',
                     side_effect=ConnectionResetError)
  def test_adapter(self, mocked):

    s = requests_extensions.HookableSession()
    try:
      s.mount('http://', adapters.HTTPAdapter(max_retries=urllib3.Retry(total=1, connect=1)))
      s.request('GET', 'http://www.test.com')
    except requests.ConnectionError:
      pass

    self.assertEqual(mocked.call_count, 2)
user_79505890
Related