Google Storage - Read timeout

I am using signed url of in Google Storage. I have a python code using requests that upload a file to the signed url using http PUT. The code is running in a GKE 1.17 deployment with 18 threads on ubuntu 18.04. The code implements retry using the retry python library. In addition the code is crawling websites.

The Problem

The code sometimes fail with the exception ReadTimeout:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1346, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 874, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 423, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
...

    response = self.__upload(target, file_stream, headers)
  File "<decorator-gen-18>", line 2, in __upload
  File "/usr/local/lib/python3.6/dist-packages/retry/api.py", line 74, in retry_decorator
    logger)
  File "/usr/local/lib/python3.6/dist-packages/retry/api.py", line 33, in __retry_internal
    return f()
  File "/home/crawler/crawler/upload/UploadResults.py", line 179, in __upload
    target, zip_file_stream, headers=headers, timeout=TIMEOUT)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 593, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Read timed out. (read timeout=30)

This means that the connection was made successfully and no data was received in the readtimeout window.

My Attempts To Fix It

1. More retries - Up to 10
2. Larger delay between retries
3. exponential backoff
4. Test if I raise ReadTimeout() in the function if the functions retries this specific exception, and it did.
5. Allow more connections in for the session and allow more connections per host.
6. The google storage url is avoiding naming bottleneck by using a different folder for each file.
7. Larger read timeout value - tried with up to 60 seconds.

My Code

import requests
from google.api_core.exceptions import GatewayTimeout, GoogleAPICallError, ServiceUnavailable
from requests.exceptions import ConnectionError
from crawler.exceptions.CrawlerException import CrawlerException, eCrawlingStatusCodes
from retry import retry

RETRIES = 10
BACKOFF = 1.5
DELAY = 2
TIMEOUT = (9.05, 30)

class UploadResults:
    GOOGLE_STORAGE = GoogleStorage()
    SESSION = requests.Session()
    SESSION.mount('https://', requests.adapters.HTTPAdapter(
        pool_connections=20, pool_maxsize=30))

    @retry(exceptions=(CrawlerException, requests.exceptions.RequestException, ValueError, 
            requests.exceptions.Timeout, requests.exceptions.ReadTimeout, requests.exceptions.ConnectTimeout),
           tries=RETRIES, delay=DELAY, backoff=BACKOFF)
    def __upload(self, target, zip_file_stream, headers):
        response = UploadResults.SESSION.put(
            target, zip_file_stream, headers=headers, timeout=TIMEOUT)

        if response.status_code >= 300:
            raise CrawlerException(eCrawlingStatusCodes.InternalError,
                                   "Internal error on uploading to path, Server status: %d" % response.status_code)

        return response

Questions

What can cause this exception?

Is there a limit I am missing that can cause this behaviour?