Hitting TooManyRequests 429 when using GoogleCloudStorageComposeOperator

user_524293 0
Viewed 232

When using GoogleCloudStorageComposeOperator in Google's Cloud Composer we've started hitting TooManyRequests, HTTP 429.

The rate of change requests to the object path/file.csv exceeds the rate limit. Please reduce the rate of create, update, and delete requests.

What limit are we hitting? I think it's this limit but I'm not sure:

There is a write limit to the same object name of once per second, so rapid writes to the same object name won't scale.

Does anyone have a sane way around this issue? On retry it usually works, but would be neat to not rely on it working on retry.

google-cloud-composer airflow google-cloud-storage
1 Answers

It's hard to say, without details, but this is rather Storage than Composer Issue. It is described in Troubleshooting guide for Cloud Storage.

There you can find some more references to dig more about it. On Quotas and Limit page I have found:

When a project's bandwidth exceeds quota in a location, requests to affected buckets can be rejected with a retryable 429 error or can be throttled. See Bandwidth usage for information about monitoring your bandwidth.

It seems that this error is intended to be retried, so I think implementation of try/catch mechanism might be a solution.

user_124103320
Related