ls in awscli returns "PRE". Why and how to get rid of it

user_8236936 0
Viewed 11435

Using awscli in git bash, the command 

aws s3 ls "s3://directory/"

returns a list of

PRE "filename"

This is inconvenient as I need to do further commands on the output and I only need the file/directory names within the given directory.

For instance, it would be nice to be able to do:

for dir in $(aws s3 ls s3://directory/) do
 aws s3 ls $dir | grep .json;
done

Any suggestions to work around this?

aws-cli git-bash ls amazon-s3
3 Answers

What is "PRE"?

You can think of prefixes as a way to organize your data in a similar way to directories. However, prefixes are not directories.

How do I get rid of "PRE" in the output?

  • Use the aws s3api list-objects-v2 command which supports querying and output formatting instead of the aws s3 ls command.
aws s3api list-objects-v2 --bucket <mybucket_name> --prefix <path> --query "Contents[?contains(Key, '.json')].Key"

Note that <bucketname> here is just the name of the bucket. Do not include s3:// or any additional /.

The --query parameter is what gives you the powerful output.

For example

  • --query Contents[].Key would print all of the keys.
  • --query Contents[?contains(Key, '.json')].Key specifies additional criteria on the Key (the Key contains .json (e.g. the extension)) and then the final .Key tells it what attribute of the json output to return.
user_24291760

To list all folders:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep "\/$" | sed 's/\/$//'

To list all files:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep -v /$

To list all .json files:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep "\.json$"

To list all .json and all .yaml files:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep -E "(\.yaml|\.json)$"

To list all files except .json files:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep -v "\.json$"
user_2714150
Related