Python Google Cloud Function Connection reset by peer

Question

As detailed here: https://issuetracker.google.com/issues/113672049

Cross-posted here: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5879)

I'm getting a connection reset error when using the Firebase Storage API from a Google Cloud Function in Python.

The deployed function is calling one blob-get i.e.

from firebase_admin import storage

def fn(request):
  bucket = 'my-firebase-bucket'
  path = '/thing'
  blob = storage.bucket(bucket).get_blob(path)

The failure is intermittent; the function has around a 90% success rate.

It seems more likely to fail the first time the function is called after it is deployed.

user_5525118 · Answer

Cloud functions are stateless, but can re-use global state from previous invocations. This is explained in tips and these docs.

Using global state with retries should give you a more robust function:

from tenacity import retry, stop_after_attempt, wait_random
from firebase_admin import storage

@retry(stop=stop_after_attempt(3), wait=wait_random(min=1, max=2))
def get_bucket(storage):
    return storage.bucket('my-firebase-bucket')

@retry(stop=stop_after_attempt(3), wait=wait_random(min=1, max=2))
def get_blob(bucket, path):
    return bucket.get_blob(path)

bucket = get_bucket(storage)

def fn(request):
  path = '/thing'
  blob = get_blob(bucket, path)
  # etc..

user_1236326 · Answer

You may want to check how many clients you are creating.

Try to reuse network connections across function invocations, as described in Optimizing Networking. However, note that a connection that remains unused for 2 minutes might be closed by the system, and further attempts to use a closed connection will result in a "connection reset" error. Your code should either use a library that handles closed connections well, or handle them explicitly if using low-level networking constructs.

https://cloud.google.com/functions/docs/concepts/exec#network

See this example where they only create a client once and reuse it in functions:

import os
from google.cloud import pubsub_v1

# Create a global Pub/Sub client to avoid unneeded network activity
pubsub = pubsub_v1.PublisherClient()


def gcp_api_call(request):
    """
    HTTP Cloud Function that uses a cached client library instance to
    reduce the number of connections required per function invocation.
    Args:
        request (flask.Request): The request object.
    Returns:
        The response text, or any set of values that can be turned into a
        Response object using `make_response`
        .
    """

    project = os.getenv('GCP_PROJECT')
    request_json = request.get_json()

    topic_name = request_json['topic']
    topic_path = pubsub.topic_path(project, topic_name)

    # Process the request
    data = 'Test message'.encode('utf-8')
    pubsub.publish(topic_path, data=data)

    return '1 message published'

https://cloud.google.com/functions/docs/bestpractices/networking#accessing_google_apis

Python Google Cloud Function Connection reset by peer

2 Answers