How do I read files in a subdirectory/prefix from google cloud storage?

user_13070421 0
Viewed 1013

I have moved files and folders over to google cloud storage (GCS). I am finding it difficult to understand the prefixes and delimiters in the GCS documentation.

What I want to do is essentially replace the path/location of locally stored files with GCS location. E.g.

It is currently coded for local path:

Variable = "C:\\Users\\admin\\Documents\\Folder1\\doc3.csv"

I need it written to search for the file in GCS like below:

Variable = "https://storage.cloud.google.com/MYBUCKETNAME/Folder1/doc3.csv?supportedpurview=project"

This obviously doesn't work but I have the below code which connects to the bucket but I am struggling to direct it to the specific file.

from google.cloud import storage
import os
client = storage.Client()
bucket = client.bucket('mybucketname')
blobs = bucket.list_blobs(prefix='Folder1')
for blob in blobs:
    print(blob.name)

So the output of this gives the following files in that specific folder:

doc1.csv
doc2.csv
doc3.csv

For my variable, what do I write as my filepath to doc3.csv. This is what I am struggling with.

google-cloud-storage google-cloud-platform prefix python
1 Answers

You can't refer to a blob in Cloud Storage by it's name like it's a local file. You'll need to transfer the file from Cloud Storage to your local filesystem first:

destination_file_name = ...
blob.download_to_filename(destination_file_name)

Then you can read the file from where you've stored it locally:

with open(destination_file_name) as f:
   contents = f.read()

...or otherwise use it like any other local file on your filesystem.

See https://cloud.google.com/storage/docs/downloading-objects for more details.

user_3280360
Related