For some reason there's a bucket with a bunch of different files, all of which have the same prefix but with different dates:
backup.2017-01-01aa
backup.2017-01-01ab
backup.2017-01-15aa
backup.2017-01-15ab
backup.2017-02-01aa
backup.2017-02-01ab
etc..
How do I download only files that start with "backup.2017-01-01"?
I think --include does the filtering locally. So if your bucket contains millions of files, the command can take hours to run, because it needs to download a list of all the filenames in the bucket. Also, some extra network traffic.
But aws s3 ls can take a truncated filename to list all the corresponding files, without any extra traffic. So you can
aws s3 ls s3://yourbucket/backup.2017-
to see your files, and something like
aws s3 ls s3://yourbucket/backup.2017- | colrm 1 31 | xargs -I % aws s3 cp s3://yourbucket/% .
to copy your files.
A PowerShell equivalent for @sampo-smolander's answer
,@(Get-ChildItem -recurse | aws s3 ls s3://yourbucket/backup.2017-) | Select-Object -ExpandProperty syncroot | foreach-Object {$_.split(" ")[-1]} | %{&"aws" s3 cp s3://yourbucket/$_ .}
Posting it here since I spent a lot of time figuring out this, so hopefully it helps someone else who needs to use powershell. Also I'm not too familiar with powershell so it may need optimization.