how to get list of files on aws bucket with a specific pattern

user_654019 0
Viewed 1950

I am looking for a way to get the list of files with a specific pattern from aws. But it seems that it can not support it.

In detail:

I have a bucket in AWS that has several hundred thousand files in it and I am want to find that has a specific pattern. For example, I want to find all files that are "my*.txt" on the bucket and list them.

If I run this command:

aws s3 ls s3://my_bucket/  --recursive > byBucketList.txt

Then I can search for my files in the text file and found where are they. But I am not able to run anything such as this:

 aws s3 ls s3://my_bucket/my*.txt  --recursive > byBucketList.txt

to only get the list of files that I am looking for.

Is there any way that I get the list of files with a specific pattern on an AWS bucket?

amazon-web-services amazon-s3
2 Answers

Sadly you can't do this with s3 ls. But you could possibly use Exclude and Include Filters along with --dryrun for commands that support filters. For example, for s3 cp:

aws s3 cp s3://my_bucket ./. --recursive --dryrun --exclude "*" --include "my*.txt"

--dryrun (boolean) Displays the operations that would be performed using the specified command without actually running them.

This should print out all the objects that would be normally copied without --dryrun option.

Since you have a large bucket, you can test it out on small bucket just to get the feel of the command as its output is different then from s3 ls.

user_2488230

mans you can achieve this using grep.

 aws s3 ls s3://my_bucket --recursive | grep -e 'my[A-Za-z0-9]*.txt' > byBucketList.txt
user_44533510
Related