We've set up our deployments to run using Workload Identity. Each deployment has its own technical identity, a GCP service account set up with the required IAM roles, and a Kubernetes service account to match (both linked as described in the Workload Identity documentation).
This all works just fine, with one small problem: almost every time a pod starts it gives this error:
Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. at GoogleAuth.getApplicationDefaultAsync (/home/node/node_modules/google-auth-library/build/src/auth/googleauth.js:155:19) at processTicksAndRejections (internal/process/task_queues.js:97:5) at async GoogleAuth.getClient (/home/node/node_modules/google-auth-library/build/src/auth/googleauth.js:486:17) at async GrpcClient._getCredentials (/home/node/node_modules/google-gax/build/src/grpc.js:88:24) at async GrpcClient.createStub (/home/node/node_modules/google-gax/build/src/grpc.js:213:23)
It then crash-loops and typically comes up later, after 2 or 3 retries. Very very odd...
To make matters worse, every now and then we get this error instead:
Error: 403 undefined: Getting metadata from plugin failed with error: Could not refresh access token: A Forbidden error was returned while attempting to retrieve an access token for the Compute Engine built-in service account. This may be because the Compute Engine instance does not have the correct permission scopes specified: Could not refresh access token: Unsuccessful response status code. Request failed with status code 403 at Object.callErrorFromStatus (/home/node/node_modules/@grpc/grpc-js/build/src/call.js:31:26) at Object.onReceiveStatus (/home/node/node_modules/@grpc/grpc-js/build/src/client.js:176:52) at Object.onReceiveStatus (/home/node/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:342:141) at Object.onReceiveStatus (/home/node/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181) at /home/node/node_modules/@grpc/grpc-js/build/src/call-stream.js:124:78 at processTicksAndRejections (internal/process/task_queues.js:79:11)
And when we see this, it's basically game over until we try all kinds of black voodoo to try and get the service back up (usually it boils down to us deleting the entire auth stuff for that service and recreating things from scratch).
Anyone has any idea what could be going on here?
Our cluster is running 1.16.13-gke.1, and has Istio 1.6.7 installed (using the Istio operator).