Error 429: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user'

user_8823763 0
Viewed 1236

We migrated a service from GCP App Engine to Cloud Run. This service is connected to Cloud SQL. enter image description here

and its parameters are:

  • max requests per container: 1
  • min instances: 0
  • max instances: 200

After receiving spikes, we have experienced this error multiple times:

Cloud SQL connection failed. Please see https://cloud.google.com/sql/docs/mysql/connect-overview for additional details: googleapi: Error 429: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'sqladmin.googleapis.com' for consumer 'project_number:370240628xxx'., rateLimitExceeded

The quota Queries per minute per user in Cloud SQL Admin API is set to 180. So this is obviously the point of failure.

I found out that the quota limit is hit due to the way how Cloud Run is connected to the Cloud SQL instance. As this link says:

The native connection leverages the Cloud SQL Auth proxy which in turns makes requests to the Cloud SQL Admin API. This API - like all GCP APIs - has throttling quotas which will prevent containers from starting once reached.

The tutorial on the same website suggests using VPC connector instead of the normal Cloud SQL connection. But I've seen Google's docs and other stackoverflow posts suggesting adding quotaUser parameter somehow.

What would be the optimal solution? I would normally go for the VPC but since that is a paid service, I try to avoid that obviously. Since I already pay for Cloud Run and Cloud SQL, I expect those two to be connected without issues and additional costs.

Thank you for all your tips and help!

google-cloud-run google-cloud-sql knex.js docker google-cloud-platform
2 Answers

max requests per container: 1
min instances: 0
max instances: 200

Wow, that's a strange setup.

The above means you have concureny=1. And you have for each request a brand new container firing up. All your request have slow starts.

You should increase max requests per container up to a threshold that your container based on the process that it runs can keep up, eg: 80 request/container for a simple webservice with 512MB sounds good. Experiment with this.

Increasing concurrency from 1 to 80, it means 1 single database connection can be reused, and you are not hitting the other limits.

ps. I am not aware of any method that you can specify quotaUser for the SQL connection, which is a unix socket type connection.

user_2437820

Short explanation:

We solved the issue by using the VPC serverless connector (VPC network -> Serverless VPC access) that allows us to create an internal network.

The quota "Queries per minute per user" is for Cloud SQL Admin API. Since we use an internal network now, we don't hit this API and its limits.

Steps:

  1. Register a new VPC serverless connector. Can be found in VPC network -> Serverless VPC access. We chose the "default" network, set the IP address range to "10.8.0.0/28" and "e2-micro" as an instance type. Regarding the region, we specified "europe-west1" since our Cloud Run services are based there. As Google documentation says, the VPC's region must match your services' region to work.
  2. You must enable "Private IP" for your SQL. It can be found in SQL -> Connections. We chose the "default" network and chose Google to generate the address for us. It took a few seconds for Google to finish this process.
  3. Then, to set these settings, you have to press the "SAVE" button, BUT be careful as the process that follows takes a few minutes to complete (7min in our case)
  4. We went to SQL -> Overview and copied the newly generated "Private IP address".
  5. Then, we used this address as an environment variable in our Cloud Run as SQL_HOST. Before, the SQL_HOST variable was "Connection name" instead, which had this form "project-name:europe-west3:database-name". So this was replaced with the "Private IP".

It worked, although we think Google should do this by default. Cloud Run is supposed to be a service that can be scaled to high numbers, but you quickly find out that there are issues resulting from the increasing traffic.

UPDATE:

We ended up creating a new VPC network just for the connection from Cloud Run services to the database. So instead of "default", we assigned our database and the serverless VPC connector to this newly created network. This way we can easily modify how many IP addresses we want to allow without interfering with other machines in the network.

user_88237630
Related