Amazon S3 how to list files in a “folder”

Viewed 15846

I set the key of files in Amazon S3 to be folder\filename. Is there a way to get all the files under a "folder" (search files by regex)?

3 Answers

You tagged your question with aws-sdk but did not mention a language, so I'll use Python in this answer.

The list_objects_v2() command accepts a Prefix:

response = client.list_objects_v2(
    Bucket='string',
    Delimiter='string',
    EncodingType='url',
    MaxKeys=123,
    Prefix='string',
    ContinuationToken='string',
    FetchOwner=True|False,
    StartAfter='string',
    RequestPayer='requester'
)

If you set Prefix='folder/', then it will return objects within that folder.

However, it is not possible to use a Regex expression. Your program will need to filter the return list to meet your needs.

I hope provided link will answer your question.

AWS S3 object listing

You can also get list of objects by using aws-cli

Type following command in terminal

aws s3 ls bucketName/folderName/

Here '/' is necessary at the end of folder name, else you will get only folder name in result.

If you want to get the list of files in a SUBFOLDER that is present across folders in a S3 bucket, we can do it using list_objects API. Say, you need all files in abc subfolder of bucket test having following files:

folderA/abc/fileabcX
folderA/def/filedefX
folderB/abc/fileabcY
folderB/def/filedefY
folderC/abc/fileabcZ
folderC/def/filedefZ
folderC/xyz/filexyzZ

In Python, we can do it by calling list_objects_v2 API twice, where you get the folder the first time using empty prefix and / delimiter and using "CommonPrefixes" in the output. Then, call list_objects_v2 again with your subfolder as part of the prefix with the folder name that you get in the first call:

import boto3
client = boto3.client('s3')
objs = client.list_objects_v2(Bucket="test", Prefix='', Delimiter='/')
for prefix in objs.get('CommonPrefixes')
    folder = prefix.get('Prefix')
    resp = client.list_objects_v2(
        Bucket="test",
        Prefix=folder + 'abc/',
    )
    if resp.get('Contents') is not None:
        for obj in resp.get('Contents'):
            print(obj.get('Key'))

Output:

fileabcX
fileabcY
fileabcZ
Related