gcs bucket how to get objects using filters [golang]

nonus25 0
Viewed 50

Is there a way to get list of objects which starts and ends with specific criteria as example:

a/b/c/id1.json
a/b/c/id2.json
a/b/c/id3.json
a/c/id1.json

and we wanna query for

Prefix: "a/",
EndOffset: "id1.json"

expected output should be:

a/b/c/id1.json

and we wanna filter out other options and we don't know what the b folder name would be.

So:

a is always a

b is random uniq string

c is always c

and we always want the specific json.

As i am tying to achieve this with:

query := &storage.Query{
        Prefix: "a/",
        //StartOffset: "",
        EndOffset: "id1.json",
        //Delimiter:
}
query.SetAttrSelection([]string{"Name"})

or

Prefix:                   "c/id1.json",
Delimiter:                "/",
IncludeTrailingDelimiter: true,

and for some reason i am getting in return all of those files.

And of course i would like to limit the results as much as possible for better performances.

Maybe there is a way to use some regex in Prefix definition ? like a/*/c/id1.json

Thanks

----------------------- ========= Edited ========= -----------------------

Please note that this is already implemented by me storage_list_files_with_prefix-go and do not work as i would like to have it. So the main question is HOWTO make this filtering working with the example I am showing.

google-cloud-storage go google-cloud-platform
1 Answers

Key points:

  • Cloud Storage Buckets do not have directories.
  • The namespace is flat.
  • Object names are just strings.
  • The slash / character which is often used to separate directory names in file systems is just a character in an Bucket object name. The slash has no significance but can be used as a delimiter.
  • You can specify a prefix and a delimiter to reduce the returned object list.
  • Cloud Storage does not support regex expressions.
  • The asterisk * is a character and not a wildcard.

Summary:

  • You must implement additional filtering in your code.

List the objects in a bucket using a prefix filter

John Hanley0
Related