Accessing Google Secret Manager from AI Platform training job with custom container

I am trying to access a secret stored in Google Secret Manager from an AI Platform Training job that runs in a custom container. I am using the following Python code to retrieve secrets:

# Standard library imports
import os

# Import the Secret Manager client library.
from google.cloud import secretmanager


def access_secret_version(secret_id, version_id, project_id=os.environ.get("GCP_PROJECT")):
     # Create the Secret Manager client.
    client = secretmanager.SecretManagerServiceClient()

    # Build the resource name of the secret version.
    name = client.secret_version_path(project_id, secret_id, version_id)

    # Access the secret version.
    response = client.access_secret_version(name)

    # Return the secret payload.
    return response.payload.data.decode("UTF-8")

I have added the Secret Manager Secret Accessor role to the default Cloud ML service account (service-XXX@cloud-ml.google.com.iam.gserviceaccount.com), but access is still denied:

google.api_core.exceptions.PermissionDenied: 403 Request had insufficient authentication scopes.

Have I granted the correct role to the correct service account or I am missing something else?