I set the key of files in Amazon S3 to be folder\filename. Is there a way to get all the files under a "folder" (search files by regex)?
I set the key of files in Amazon S3 to be folder\filename. Is there a way to get all the files under a "folder" (search files by regex)?
You tagged your question with aws-sdk but did not mention a language, so I'll use Python in this answer.
The list_objects_v2() command accepts a Prefix:
response = client.list_objects_v2(
Bucket='string',
Delimiter='string',
EncodingType='url',
MaxKeys=123,
Prefix='string',
ContinuationToken='string',
FetchOwner=True|False,
StartAfter='string',
RequestPayer='requester'
)
If you set Prefix='folder/', then it will return objects within that folder.
However, it is not possible to use a Regex expression. Your program will need to filter the return list to meet your needs.
I hope provided link will answer your question.
You can also get list of objects by using aws-cli
Type following command in terminal
aws s3 ls bucketName/folderName/
Here '/' is necessary at the end of folder name, else you will get only folder name in result.
If you want to get the list of files in a SUBFOLDER that is present across folders in a S3 bucket, we can do it using list_objects API. Say, you need all files in abc subfolder of bucket test having following files:
folderA/abc/fileabcX
folderA/def/filedefX
folderB/abc/fileabcY
folderB/def/filedefY
folderC/abc/fileabcZ
folderC/def/filedefZ
folderC/xyz/filexyzZ
In Python, we can do it by calling list_objects_v2 API twice, where you get the folder the first time using empty prefix and / delimiter and using "CommonPrefixes" in the output. Then, call list_objects_v2 again with your subfolder as part of the prefix with the folder name that you get in the first call:
import boto3
client = boto3.client('s3')
objs = client.list_objects_v2(Bucket="test", Prefix='', Delimiter='/')
for prefix in objs.get('CommonPrefixes')
folder = prefix.get('Prefix')
resp = client.list_objects_v2(
Bucket="test",
Prefix=folder + 'abc/',
)
if resp.get('Contents') is not None:
for obj in resp.get('Contents'):
print(obj.get('Key'))
Output:
fileabcX
fileabcY
fileabcZ