Listing Blobs in GCS using java with pagination keep getting the same page of blobs

user_8099998 0
Viewed 1258

I am trying to list a large amount of gcs blobs using the Java API. Since the number of blobs is large, I tried to use pagination, but I am getting the same page repeatedly. The code looked like this:

Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();

Page<Blob> allBlobs = storage.list(myBucketName,Storage.BlobListOption.pageSize(5000), Storage.BlobListOption.prefix("some prefix");

while (allBlobs.hasNextPage()) {
   Page<Blob> page = allBlobs.getNextPage();
   for (Blob blob : page.getValues()) {
     .... do something....
   }

}

It looks like I am getting the same page over and over. I looked at the token by allBlobs.getNextPageToken() and the token looked the same all the time. Did I miss something to move the page forward to the next one? Isn't getNextPage doing it? The Page interface has only a few methods defined. Did I overlook something?

google-cloud-storage google-cloud-platform java
2 Answers

Use the iterateAll method instead. See example here (copied here for completion):

Page<Blob> blobs =
    storage.list(
        bucketName, BlobListOption.currentDirectory(),
        BlobListOption.prefix(directory));

for (Blob blob : blobs.iterateAll()) {
  // do something with the blob
}
user_6249000

I haven't worked with the Java, but is it possible that getNextPage() is returning the next page, but not iterating allBlobs? So allBlobs' position in the storage list isn't changing? What happens if you add a line after your for loop that does "allBlobs = page";

user_35346900
Related