429 Rate exceeded responses for no apparent reason

Question

We're using Contentful to manage CMS content. When you save content in Contentful it sends webhooks for a service we've set up on Cloud Run, which in turn ensures the updated content is built and deployed.

This setup has been previously so that the Cloud Run service was limited to 1 container max, with 80 concurrent requests limit. This should be plenty for the few webhooks we get occasionally.

Now when debugging complaints about content not being updated I bumped into a very persistent and irritating issue - Google Cloud Run does not try to process the 2 webhooks sent by Contentful, but instead responds to one of the 2 with status 429 and Rate exceeded. in response body.

This response does not come from our backend, I can see in the Cloud Run Logs tab the message generated by Google: The request was aborted because there was no available instance.

I've tried:

Increasing number of processes on the container from 1 to 2 - should not be necessary due to use of an async framework
Increasing number of containers from 1 to 2

The issue persists for the webhooks from Contentful.

If I try making requests from my local machine with hey that defaults to 200 requests with 50 concurrency, they all go through without any 429 status codes returned.

What is going on that generates 429 status codes when a specific client - in this case Contentful - makes ONLY 2 requests in quick succession? How do we disable or bypass this behavior?

gcloud run services describe gives me these details of the deployment:

+ Service [redacted] in region europe-north1

URL:     https://[redacted].a.run.app
Ingress: all
Traffic:
  100% LATEST (currently [redacted])

Last updated on 2021-01-19T13:48:46.172388Z by [redacted]:
  Revision [redacted]
  Image:            eu.gcr.io/[redacted]/[redacted]:c0a2e7a6-56d5-4f6f-b241-1dd9ed96dd30
  Port:             8080
  Memory:           256Mi
  CPU:              1000m
  Service account:  [redacted]-compute@developer.gserviceaccount.com
  Env vars:
    WEB_CONCURRENCY 2
  Concurrency:      80
  Max Instances:    2
  Timeout:          300s

user_3036129 · Answer

This is more a speculation that an answer, but I would try re-deploying you Cloud Run service with min-instances set to 1 (or more).

Here is why.

In the Cloud Run troubleshooting docs they write (emphasis mine):

This error can also be caused by a sudden increase in traffic, a long container startup time or a long request processing time.

Your Cloud Run service receives webhook events from a CMS (Contentful). And, as you wrote, these updates are rather sporadic. So I think that your situation could be the same as the one described in this comment on Medium:

I tested “max-instances: 2” and the conclusion is I got 429 — Rate exceeded responses from Google frontend proxy because no container was running. It seems that a very low count of instances will deregister your service completely from the load-balancer until a second request was made.

If Google Cloud did indeed de-register your Cloud Run service completely because it was not receiving any traffic, re-deploying the service with at least one container instance could fix your issue. Another way would be to call your Cloud Run service every once in a while just to keep it "warm".

429 Rate exceeded responses for no apparent reason

1 Answers