Roles Required to write to Cloud Storage (GCP) from python (pandas)

El Burro 0
Viewed 40

I have a question for the GCP connoisseurs among you.

I have an issue that I can upload to a bucket via UI and gsutil - but if I try to do this via python

df.to_csv('gs://BUCKET_NAME/test.csv')

I get a 403 insufficient permission error.

My guess at the moment is that python does this via an API and requires an extra role - to make things more confusing I am already project owner of the project of the bucket and compared to other team members did not really find lacking permissions for this specific bucket.

I use python 3.9.1 via pyenv and pandas '1.4.2'

Anyone had the same issue/ knows what role I am missing?

  1. I checked that I have in principal rights to upload both via UI and gsutil
  2. I used the same virtual python environemnt to read and write from bigquery to check that I can in principle use GCP data in python - this works
  3. I have the following Roles on the Bucket Storage Admin, Storage Object Admin, Storage Object Creator, Storage Object Viewer
google-cloud-storage google-cloud-platform pandas python
1 Answers

gsutil and gcloud share credentials.

These credentials are not shared with other code running locally.

The quick-fix but sub-optimal solution is to:

gcloud auth application-default login

And run the code again.

It will then use your gcloud (gsutil) user credentials configured to run as if you were using a Service Account.

These credentials are stored (on Linux) in ${HOME}/.config/gcloud/application_default_credentials.json.

A better solution is to create a Service Account specifically for your app and grant it the minimal set of IAM permissions that it will need (BigQuery, GCS, ...).

For testing purposes (!) you can download the Service Account key locally.

You can then auth your code using Google's Application Default Credentials (ADC) by (on Linux):

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/key.json
python3 your_app.py

When you deploy code that leverages ADC to a Google Cloud compute service (Compute Engine, Cloud Run, ...), it can be deployed unchanged because the credentials for the compute resource will be automatically obtained from the Metadata service.

You can Google e.g. "Google IAM BigQuery" to find the documentation that lists the roles:

DazWilkin0
Related