In vertex AI pipelines we have created a custom component that trigger a cloud run. In order to do so we need to fetch the Id Token Credentials of the running identity. This work most of the time but fails randomly every 5 to 10 hours.
here is our code :
auth_req = google.auth.transport.requests.Request()
id_token = google.oauth2.id_token.fetch_id_token(auth_req, url_dbt_server)
return id_token
the error message is:
google.auth.exceptions.RefreshError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=<Our audience>&format=full from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Not Found\\n'", <google.auth.transport.requests._Response object at 0x7f8f0d78aca0>)
How can I solve (or at least go around) this problem?