I've got a custom cloud build which runs a docker image hosted in our google container registry and is not able to connect to our Cloud SQL hosted postgres w/ SSL auth and public IP without a time out (1800 s).
The docker container has the correct client keys and root cert as well as password to connect to the Cloud SQL read replica, as I've tested this build locally with the same docker run command, and it can connect correctly (using a whitelisted IP from the computer running the docker build locally). However, when running on cloud build with the default cloud build service account, it currently always time out and is not able to connect to cloud sql.
What I've Tried
- Enabling Cloud SQL Viewer or Cloud SQL Editor permission on the cloud build service account
- Double / triple / quadruple checking all values being used to open the postgres connection to Cloud SQL
- Rotated certs/keys for Cloud SQL
- I've tried setting up a VPC for Cloud SQL and all of our services should be on the same network, (default), and pointing the container's connection IP for postgres to the private network IP, and this still does not work (still timeout error)
- Rebuilding the docker image in the cloud build step in case there was some difference in my local docker image and the one on cloud build
- Strangely, this used to work last Friday, and nothing in our code base or Postgres details have changed, so we suspect something changed on GCE's end.
- Setting Cloud SQL to allow all connections (0.0.0.0/0) works as expected, pointing to a likely issue with the service account not being setup with the correct access to Cloud SQL (Note I have set the service account's permissions to have either Cloud SQL Viewer or Cloud SQL Editor, and both still result in timeout)