How to create csv file using service account in google colab(Python) tool

user_12331179 0
Viewed 137

I want to create CSV file from pandas data frame in a google storage bucket using colab tool. Right now we use our gmail authentication to load csv file in storage bucket using below command

df.to_csv("gs://Jobs/data.csv")

I have checked below google links https://cloud.google.com/docs/authentication/production#linux-or-macos

Currently, we used the below code to get credentials from service account

 def getCredentialsFromServiceAccount(path: str) -> service_account.Credentials:
    return service_account.Credentials.from_service_account_info(
        path
    )

Kindly suggest

service-accounts google-colaboratory google-drive-api python
1 Answers

There are two main approaches you can use to upload a csv to Cloud Storage within Colab. Both store the csv locally first and then upload it.

  1. Use gsutil from within Colab.
  2. Use Cloud Storage Python Client Library

The first approach is easiest but authenticates with the user signed into Colab and not a service account.

from google.colab import auth

auth.authenticate_user()

bucket_name = 'my-bucket'
df.to_csv('data.csv', index = False)
!gsutil cp 'data.csv' 'gs://{bucket_name}/'

The next approach uses the client library and authenticates with a service account.

from google.cloud import storage

bucket_name = 'my-bucket'

# store csv locally
df.to_csv('data.csv', index = False)
# start storage client
client = storage.Client.from_service_account_json("path/to/key.json")
# get bucket
bucket = client.bucket(bucket_name)
# create blob (where you want to store csv within bucket)
blob = bucket.blob("jobs/data.csv")
# upload blob to bucket
blob.upload_from_filename("data.csv")
user_166086270
Related