Quick way to list all files in Amazon S3 bucket?

Be carefull, amazon list only returns 1000 files. If you want to iterate over all files you have to paginate the results using markers :

In ruby using aws-s3

bucket_name = 'yourBucket'
marker = ""

AWS::S3::Base.establish_connection!(
  :access_key_id => 'your_access_key_id',
  :secret_access_key => 'your_secret_access_key'
)

loop do
  objects = Bucket.objects(bucket_name, :marker=>marker, :max_keys=>1000)
  break if objects.size == 0
  marker = objects.last.key

  objects.each do |obj|
      puts "#{obj.key}"
  end
end

end

Hope this helps, vincent

There are couple of ways you can go about it. Using Python

import boto3

sesssion = boto3.Session(aws_access_key_id, aws_secret_access_key)

s3 = sesssion.resource('s3')

bucketName = 'testbucket133'
bucket = s3.Bucket(bucketName)

for obj in bucket.objects.all():
    print(obj.key)

Another way is using AWS cli for it

aws s3 ls s3://{bucketname}
example : aws s3 ls s3://testbucket133

First make sure you are on an instance terminal and you have all access of S3 in IAM you are using. For example I used an ec2 instance.

pip3 install awscli

Then Configure aws

aws configure

Then fill outcredantials ex:-

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json (or just press enter)

Now, See all buckets

aws s3 ls

Store all buckets name

aws s3 ls > output.txt

See all file structure in a bucket

aws s3 ls bucket-name --recursive

Store file structure in each bucket

aws s3 ls bucket-name --recursive > file_Structure.txt

Hope this helps.

For Python's boto3 after having used aws configure:

import boto3
s3 = boto3.resource('s3')

bucket = s3.Bucket('name')
for obj in bucket.objects.all():
    print(obj.key)

AWS CLI can let you see all files of an S3 bucket quickly and help in performing other operations too.

To use AWS CLI follow steps below:

1. Install AWS CLI.
2. Configure AWS CLI for using default security credentials and default AWS Region.

3. To see all files of an S3 bucket use command

   aws s3 ls s3://your_bucket_name --recursive

Reference to use AWS cli for different AWS services: https://docs.aws.amazon.com/cli/latest/reference/

The below command will get all the file names from your AWS S3 bucket and write into text file in your current directory:

aws s3 ls s3://Bucketdirectory/Subdirectory/ | cat >> FileNames.txt

I know its old topic, but I'd like to contribute too.

With the newer version of boto3 and python, you can get the files as follow:

import os
import boto3
from botocore.exceptions import ClientError    

client = boto3.client('s3')

bucket = client.list_objects(Bucket=BUCKET_NAME)
for content in bucket["Contents"]:
    key = content["key"]

Keep in mind that this solution not comprehends pagination.

For more information: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects

Here's a way to use the stock AWS CLI to generate a diff-able list of just object names:

aws s3api list-objects --bucket "$BUCKET" --query "Contents[].{Key: Key}" --output text

(based on https://stackoverflow.com/a/54378943/53529)

This gives you the full object name of every object in the bucket, separated by new lines. Useful if you want to diff between the contents of an S3 bucket and a GCS bucket, for example.

In javascript you can use

s3.listObjects(params, function (err, result) {});

to get all objects inside bucket. you have to pass bucket name inside params (Bucket: name).

Simplified and updated version of the Scala answer by Paolo:

import scala.collection.JavaConversions.{collectionAsScalaIterable => asScala}
import com.amazonaws.services.s3.AmazonS3
import com.amazonaws.services.s3.model.{ListObjectsRequest, ObjectListing, S3ObjectSummary}

def buildListing(s3: AmazonS3, request: ListObjectsRequest): List[S3ObjectSummary] = {
  def buildList(listIn: List[S3ObjectSummary], bucketList:ObjectListing): List[S3ObjectSummary] = {
    val latestList: List[S3ObjectSummary] = bucketList.getObjectSummaries.toList

    if (!bucketList.isTruncated) listIn ::: latestList
    else buildList(listIn ::: latestList, s3.listNextBatchOfObjects(bucketList))
  }

  buildList(List(), s3.listObjects(request))
}

Stripping out the generics and using the ListObjectRequest generated by the SDK builders.

Use plumbum to wrap the cli and you will have a clear syntax:

import plumbum as pb
folders = pb.local['aws']('s3', 'ls')

please try this bash script. it uses curl command with no need for any external dependencies

bucket=<bucket_name>
region=<region_name>
awsAccess=<access_key>
awsSecret=<secret_key>
awsRegion="${region}"
baseUrl="s3.${awsRegion}.amazonaws.com"

m_sed() {
  if which gsed > /dev/null 2>&1; then
    gsed "$@"
  else
    sed "$@"
  fi
}

awsStringSign4() {
  kSecret="AWS4$1"
  kDate=$(printf         '%s' "$2" | openssl dgst -sha256 -hex -mac HMAC -macopt "key:${kSecret}"     2>/dev/null | m_sed 's/^.* //')
  kRegion=$(printf       '%s' "$3" | openssl dgst -sha256 -hex -mac HMAC -macopt "hexkey:${kDate}"    2>/dev/null | m_sed 's/^.* //')
  kService=$(printf      '%s' "$4" | openssl dgst -sha256 -hex -mac HMAC -macopt "hexkey:${kRegion}"  2>/dev/null | m_sed 's/^.* //')
  kSigning=$(printf 'aws4_request' | openssl dgst -sha256 -hex -mac HMAC -macopt "hexkey:${kService}" 2>/dev/null | m_sed 's/^.* //')
  signedString=$(printf  '%s' "$5" | openssl dgst -sha256 -hex -mac HMAC -macopt "hexkey:${kSigning}" 2>/dev/null | m_sed 's/^.* //')
  printf '%s' "${signedString}"
}

if [ -z "${region}" ]; then
  region="${awsRegion}"
fi


# Initialize helper variables

authType='AWS4-HMAC-SHA256'
service="s3"
dateValueS=$(date -u +'%Y%m%d')
dateValueL=$(date -u +'%Y%m%dT%H%M%SZ')

# 0. Hash the file to be uploaded

# 1. Create canonical request

# NOTE: order significant in ${signedHeaders} and ${canonicalRequest}

signedHeaders='host;x-amz-content-sha256;x-amz-date'

canonicalRequest="\
GET
/

host:${bucket}.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:${dateValueL}

${signedHeaders}
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

# Hash it

canonicalRequestHash=$(printf '%s' "${canonicalRequest}" | openssl dgst -sha256 -hex 2>/dev/null | m_sed 's/^.* //')

# 2. Create string to sign

stringToSign="\
${authType}
${dateValueL}
${dateValueS}/${region}/${service}/aws4_request
${canonicalRequestHash}"

# 3. Sign the string

signature=$(awsStringSign4 "${awsSecret}" "${dateValueS}" "${region}" "${service}" "${stringToSign}")

# Upload

curl -g -k "https://${baseUrl}/${bucket}" \
  -H "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
  -H "x-amz-Date: ${dateValueL}" \
  -H "Authorization: ${authType} Credential=${awsAccess}/${dateValueS}/${region}/${service}/aws4_request,SignedHeaders=${signedHeaders},Signature=${signature}"

For getting full links run

aws s3 ls s3://bucket/ | awk '{print $4}' | xargs -I{} echo "s3://bucket/{}"

This is an old question but the number of responses tells me many people hit this page.

The easiest way I found is to just use the built in AWS console for creating an inventory. It's easy to set up but the first CSV file can take up to 48 hours to show up. After that you can create either a daily or weekly output to a bucket of your choosing.