Delete millions of objects from a folder in GCS bucket

user_13756495 0
Viewed 290

I have a requirement where I need to delete millions of objects from a folder in a GCS bucket. This is my current implementation.

bucket = self.storage_client.bucket(bucket_name)
blobs = bucket.list_blobs(prefix=folder_name)
            
for blob in blobs:
    blob.delete()

As the number of objects is more, listing is not the right way. Need to find a way to do some multi-threading approach.

google-cloud-storage google-cloud-platform python
1 Answers

If you wanted to delete the entire bucket objects, Setting bucket's lifecycle to 0 would be a way to fast deletion.

But this approach is not applied when you want to delete a folder(It is same as just deleting some part of entire object).

If the folder you want to delete takes up most of the entire objcet, I think deleting entire bucket through lifecycle management and creating new bucket with remainings can be more faster way.

Edit) gsutil rm command with -m flag can multi-threding deletion. This could be a faster way too.

user_148350410
Related