How to paginate s3 bucket query using ruby gem for rails?

user_1642409 0
Viewed 174

I have a query that fetches all the items inside an S3 bucket, and it takes a very long time to load if there are a lot of items in that bucket.

bucket.objects(prefix: 'path/to/items').collect(&:key)

Based on the docs, I see it's possible to limit the number of items in that fetch.

bucket.objects(prefix: 'path/to/items').limit(10).collect(&:key)

However, I don't see any offset parameter for this function. Any idea how I could do pagination with this so it doesn't take so long to load the first few results?

amazon-web-services amazon-s3 ruby-on-rails
1 Answers

AWS provides some examples to how to do that here. It sounds like you should be able to call .each on bucket.objects(prefix: 'path/to/items') to handle each page as it is returned.

Automatic paging:

s3 = Aws::S3::Client.new

s3.list_objects(bucket: 'aws-sdk').each do |response|
  puts response.contents.map(&:key)
end

Manual paging:

s3 = Aws::S3::Client.new

# Get the first page of data
response = s3.list_objects(bucket: 'aws-sdk')

# Get additional pages
while response.next_page? do
  response = response.next_page
  # Use the response data here...
  puts response.contents.map(&:key)
end
user_3906000
Related