"Exceeded rate limits: exceeded quota for Streaming Table Creation Rate in bigquery

user_8779360 0
Viewed 86

while streaming inserts to bq I got this quota error which i couldn't find anywhere in official documentation.

com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Exceeded rate limits: Your Project: **** exceeded quota for Streaming Table Creation Rate. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors",
    "reason" : "rateLimitExceeded"
  } ],
  "message" : "Exceeded rate limits: Your Project: **** exceeded quota for Streaming Table Creation Rate. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors",
  "status" : "PERMISSION_DENIED"
}
google-bigquery
1 Answers

If you follow the URL in your error reason to a list of the error messages you might receive, you'll see that you've been rate limited.

See the rate limits for query jobs here and rate limits for API requests here.

Update

Google doesn't appear to differentiate by streaming or not, but they do cite table quotas at 1500 DML operations per table per day (not what you were doing, but still useful to note) and a max of 5 DDL operations every 10 seconds (and creating tables would fall under this bucket).

Limit your new table creations to 5 every 10 seconds or so and you shouldn't run afoul of the quotas.

user_8453560
Related