How to iterate over files in an S3 bucket?

user_995862 0
Viewed 28940

I have a large number of files (>1,000) stored in an S3 bucket, and I would like to iterate over them (e.g. in a for loop) to extract data from them using boto3.

However, I notice that in accordance with http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects, the list_objects() method of the Client class only lists up to 1,000 objects:

In [1]: import boto3

In [2]: client = boto3.client('s3')

In [11]: apks = client.list_objects(Bucket='iper-apks')

In [16]: type(apks['Contents'])
Out[16]: list

In [17]: len(apks['Contents'])
Out[17]: 1000

However, I would like to list all the objects, even if there are more than 1,000. How could I achieve this?

boto3 amazon-web-services amazon-s3 python
3 Answers
import com.amazonaws.regions.Regions
import com.amazonaws.services.s3.AmazonS3ClientBuilder
import com.amazonaws.services.s3.model.ListObjectsRequest
import java.util._

import scala.collection.JavaConverters._

val s3client = AmazonS3ClientBuilder.standard().withRegion(Regions.US_EAST_1).build()
val listObjectsRequest = new ListObjectsRequest().withBucketName("<enter_bucket_name>").withPrefix("<enter_path>").withDelimiter("/")
val bucketListing = s3client.listObjects(listObjectsRequest).getCommonPrefixes.asScala

println("")

for (file <- bucketListing) {
    println(file)
}

println("")
user_15278560
Related