I have a code that runs 24/7 (while true on the main.py file) with multiprocessing (12 workers) that accesses gcloud on a regular basis (listing buckets, downloading etc.) I dont think its a regular ssl error because I get ssl error only one time throughout a day long run and every subprocess access their own gs client. I dont really understand why the error happens or how to fix it since the same traceback gives me two types of errors. Also worked everything until i increased the amount of my workers.
The way i create my client:
def get_gs_bucket():
storage_client = storage.Client.from_service_account_json(JSON_CREDENTIALS_PATH)
gs_bucket_name = BUCKET_NAME
return storage_client.get_bucket(gs_bucket_name)
the function that's mentioned in the error:
def list_gs_dir(prefix=''):
sleep_random_prevent_gs_over_api_usage()
return gs_bucket.list_blobs(delimiter='/', prefix=prefix)._get_next_page_response()['prefixes']
my error:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.6/http/client.py", line 1373, in getresponse
response.begin()
File "/usr/lib/python3.6/http/client.py", line 311, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.6/http/client.py", line 272, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.6/ssl.py", line 874, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2309)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: <url> (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2309)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/etl/work_wish/etl/etl/utils.py", line 97, in inner
return func(*args, **kwargs)
File "/opt/etl/work_wish/etl/etl/input_bucket.py", line 119, in get_recent_sstables_group_file_per_node
sstables_ts_pairs = get_sstables_ts_pairs(node, table_type, last_ts)
File "/opt/etl/work_wish/etl/etl/input_bucket.py", line 129, in get_sstables_ts_pairs
gs_path_prefixes = list_gs_dir(prefix=node)
File "/opt/etl/work_wish/etl/etl/input_bucket.py", line 35, in list_gs_dir
return gs_bucket.list_blobs(delimiter='/', prefix=prefix)._get_next_page_response()['prefixes']
File "/usr/local/lib/python3.6/dist-packages/google/api_core/page_iterator.py", line 419, in _get_next_page_response
method=self._HTTP_METHOD, path=self.path, query_params=params
File "/usr/local/lib/python3.6/dist-packages/google/cloud/_http.py", line 315, in api_request
target_object=_target_object,
File "/usr/local/lib/python3.6/dist-packages/google/cloud/_http.py", line 192, in _make_request
return self._do_request(method, url, headers, data, target_object)
File "/usr/local/lib/python3.6/dist-packages/google/cloud/_http.py", line 221, in _do_request
return self.http.request(url=url, method=method, headers=headers, data=data)
File "/usr/local/lib/python3.6/dist-packages/google/auth/transport/requests.py", line 470, in request
**kwargs
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: <url> (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2309)'),))