Apply encryption to existing S3 objects without impacting storage class

user_8265757 0
Viewed 3390

I am trying to encrypt an existing s3 bucket. When I do this:

aws s3 cp s3://test/ s3://test/ --recursive --sse

it is encrypting all the files in the bucket by re-copying the objects. My issue here is that I have objects in the bucket in Standard, Standard-IA and Glacier storage classes. So, when I run the above copy command the objects in Standard-IA storage are converted to standard storage. (I haven't tested what happens to objects in glacier yet - probably it won't even allow me to copy.)

Is there any way where we can restore the storage type of an object and just enable encryption for an existing bucket?

amazon-glacier amazon-web-services amazon-s3 encryption
3 Answers

You could do something like this using bash and JQ, obviously python with boto3 or similar would be cleaner.

I don't know if you'd be better off adding a check to skip the GLACIER files, there's no magic way to apply encryption to them without unfreezing, then re-freezing them.

You'll want to run this on an ec2 instance local to the s3 bucket.

#!/bin/bash

bucketname="bucket-name"
aws s3 ls ${bucketname} --recursive | awk '{ print $NF }' > /tmp/filelist

for file in `cat /tmp/filelist`
do
        class=`aws s3api head-object --bucket ${bucketname} --key $file  | jq '.StorageClass' | sed 's/\"//g'`
        if [ "$class" = "null" ]
        then
                class="STANDARD"

        fi
        echo "aws s3 cp s3://${bucketname}/${file} s3://${bucketname}/${file} --sse  --storage-class ${class}"
done
user_10957420

You need to add the command line option --storage-class STANDARD_IA

user_80167200

Does your bucket have a lifecycle policy? If so - it's actually behaving like it's supposed to - you are, in effect, creating a new object in the bucket, so the transition over to standard is, in fact, correct.

The option by Ewan Leith above is really the only way to do it - programmatically determine the current storage state, then override the storage of the 'new' item on save.

Hope this helps...

user_25016000
Related