Is there a way to use the data from Google Cloud Storage directly in Colab?

user_1340852 0
Viewed 1210

I want to use a dataset (170+GB) in Google Colab. I have two questions:

  1. Since the available space in Colab is about 66GB, is there a way to use the data from GCS directly in colab, if the data is hosted in GCS? If not, what is a possible solution?

  2. How can I upload the dataset to GCS directly from a downloadable link, since I cannot wget into colab due to the limited available space?

Any help is appreciated.

google-cloud-storage google-colaboratory
1 Answers

Authenticate :

from google.colab import auth
auth.authenticate_user()

install google sdk:

!curl https://sdk.cloud.google.com | bash

init the SDK to configure the project settings.

!gcloud init

1 . Download file from Cloud Storage to Google Colab

!gsutil cp gs://google storage bucket/your file.csv .

2 . Upload file from Google Colab to Cloud

gsutil cp yourfile.csv gs://gs bucket/

Hope it helps. Source

user_98765690
Related