Authenticating a google.storage.Client without saving the service account JSON to disk

Viewed 2847

For authentication of a Google Cloud Platform storage client, I'd like to NOT write the service account JSON (credentials file that you create) to disk. I would like to keep them purely in memory after loading them from a Hashicorp Vault keystore that is shared by all cloud instances. Is there a way to pass the JSON credentials directly, rather than passing a pathlike/file object?

I understand how to do this using a pathlike/file object as follows, but this is what I want to avoid (due to security issues, I'd prefer to never write them to disk):

from google.cloud import storage

# set an environment variable that relies on a JSON file
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service_account.json"

# create the client (assumes the environment variable is set)
client = storage.Client()

# alternately, one can create the client without the environment 
# variable, but this still relies on a JSON file.
client = storage.Client.from_service_account_json("/path/to/service_account.json")

I have tried to get around this by referencing the JSON DATA (json_data) directly, but this throws the error: TypeError: expected str, bytes or os.PathLike object, not dict

json_data = {....[JSON]....}
client = storage.Client.from_service_account_json(json_data)

Also, dumping to JSON, but I get the error:

with io.open(json_credentials_path, "r", encoding="utf-8") as json_fi: OSError: [Errno 63] File name too long: '{"type": "service_account", "project_id",......

json_data = {....[JSON]....}
client = storage.Client.from_service_account_json(json.dumps(json_data))

Per the suggestion from @johnhanley, I have also tried:

from google.cloud import storage
from google.oauth2 import service_account

json_data = {...data loaded from keystore...}
type(json_data)
   dict

credentials = service_account.Credentials.from_service_account_info(json_data)
type(credentials)
   google.oauth2.service_account.Credentials

client = storage.Client(credentials=credentials)

This resulted in the DefaultCredentialsError:

raise exceptions.DefaultCredentialsError(_HELP_MESSAGE) google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://developers.google.com/accounts/docs/application-default-credentials.

If you have ideas on how to solve this, I'd love to hear it!

1 Answers

Currently there aren't built-in methods in the client library for Cloud Storage to achieve this. So here there would be 2 possibilites:

  • As @JohnHanley stated use the provided built-in methods [1][2] to create the constructor for the Cloud Storage client.

  • You might consider as well to use another product such as Cloud Functions or App Engine that would allow you to configure the authentication at the service level and avoid providing the service account credentials.

Related