Google Cloud Function The request was aborted because there was no available instance

user_517193 0
Viewed 4061

When running some cloud functions, I am getting sometimes the error:

"The request was aborted because there was no available instance."

I have seen other questions being asked with similar error but for Cloud Run where you can specify the number of instances available, but it doesnt look like there is such a thing with Cloud Function. So how to solve this problem?

I can see on the quotas page that there are limits for background functions but none for HTTP functions. I am calling the lambda function via HTTP, and it is deployed in us-central1

google-cloud-functions google-cloud-platform http
2 Answers

This issue happens both on Google Cloud Functions and Firebase Cloud Functions. We at Taskworld are facing this issue multiple times per day. People have reported reliability issue here and Google is tracking the issue here. These issues are fairly recent (created around 1–3 months after this question is posted to StackOverflow.)

We worked around this issue through chaos engineering. Specifically, we added in this code to our function HTTP endpoints:

  if (Math.random() < 0.1) {
    res.status(500).send('firebase is funny')
    return
  }

This effectively causes 10% of all requests to fail with a 500 error code which is the same behavior when Cloud Functions intrinsically cannot manage the rate of traffic. Such high error rate forces consumers to build resiliency in and add some retry logic into the client itself.

user_5599130

According to GCP dashboard, there's a currently ongoing deployment failures for both region us-central1 and europe-west1. For now there are two reasons I can think of behind the error:

  1. The ongoing issue has an indirect effect where no instances can currently handle the request.
  2. There's an initial bursts of requests coming, to the point where Cloud Functions cannot scale fast enough despite not hitting max_instances.

Solution for #1 is to temporarily re-deploy your function to another region. The solution for #2 is to implement some sort of retry logic into your function. Note that automatic retries are not available in HTTP functions, so you will have to implement it with your own logic.

If it doesn't work for you, then there's another option to implement a queuing mechanism with Cloud Tasks to handle sudden bursts of traffic and handle retries if the request fails.

user_70313080
Related