Is there a way to check if batches of files exist in GCS?

user_10768103 0
Viewed 883

I have a list of file paths that I want to check in GCS. It looks like this:

for path in all_paths:
    try:
        gcs.blob(GCS_BUCKET_NAME, path).exists()
    except google.api_core.exceptions.NotFound:
        missing_paths.append(path)

This works fine but it takes a lot of time as requests are sent one by one, for each path. Is there a way to send batches of requests in google cloud storage API ? Or any way to speed up this check ?

google-cloud-storage python-3.x
1 Answers

With Cloud Storage you can only filter by the path prefix (path/to/file.xxx). Then you will receive all the files matching this prefix even the sub path (path/to/sub/path/file.xxx). Therefore, the rest of the processing is to perform by yourselves.

And yes, if you have lot of files, it will take lot of time.

user_113725930
Related