how to copy folders from a gcp bucket folder into another bucket folder and delete it using python

user_10291435 0
Viewed 485

I want to get all folders inside a temp folder in the bucket then move it to the bucket itself and delete the temp folder, for example, if I have gs://bucketname/mainfolder/temp and this temp folder contains 3 folders like gs://bucketname/mainfolder/temp/folder1 and gs://bucketname/mainfolder/temp/folder2 and I want to move folder and folder2 (with their contents) into the mainfolder so that they are like this: gs://bucketname/mainfolder/folder1 and gs://bucketname/mainfolder/folder2 and delete the temp folder with its contents after copying what I need from it.

I was thinking about listing the folder inside the temp folder using GoogleCloudStorageHook and using list for exmaple:

hook = GoogleCloudStorageHook()
hook.list(bucket="bucketname", prefix="mainfolder/temp/", delimiter='/')

but am not so sure how to proceed after this. any ideas? :)

google-api-python-client google-cloud-storage python
1 Answers

In fact, folders doesn't exist. Therefore, you must copy files, one by one by changing their name. The name of a file (an object in fact) is the full path from the root of the bucket (because all the objects are put a the bucket level, no folder in a GCS bucket).

So, you can filter by a prefix all the objects, then change the name of the object (remove the temp) and call a copy from the old object name to the new one.

In python, you can also call the move operation, it performs the copy AND the delete just after the object copy.

Note: if you have a large number of file, it could take time (it's API call, it's slow). Think to parallelize your processing with asynchronous calls for example.

user_113725930
Related