How to get a .csv into a dataframe from gcs with credentials from script?

user_2196833 0
Viewed 2227

I've been trying to read a bucket in gcs directly into a pandas dataframe like this:

gcs_df = pandas.read_csv("gs://my_bucket/my_file.csv")

This results into:

gcsfs.utils.HttpError: Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.

due to the fact that I haven't set up any credentials in my local machine (from what I read).

For all other features of the scripts I was using a service account via:

sa_creds = service_account.Credentials.from_service_account_file("my_sa_key.json")

Can I somehow pass this info into the read_csv so I won't have to use the account in my local machine?

Any ideas?

google-cloud-storage pandas google-authentication
2 Answers

So pandas library depends on the gcsfs library. So to do the above all you have to do is the following:

import pandas
import gcsfs

fs = gcsfs.GCSFileSystem(project= <project_id>, token=<json path>)
with fs.open("gs://my_bucket/my_file.csv") as f:
    gcs_df = pandas.read_csv(f)
print(gcs_df)

The refers to the service account credentials saved as key in .json format. Based on gcsfs docs it can be a dictionary or more.

user_21968330

A bit late, but it might still be usefull for someone : If you have a path to the json file containing your credentials, you can add it to your environment variable from within your code using os.environ:

import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "my_sa_key.json"

Then everything using gcsfs should find what are now your default credentials. I tested it to write a csv with pandas and it worked.

user_149521510
Related