Is it possible to replicate a specific S3 folder between 2 buckets?

Viewed 5234

Does anyone know if it is possible to replicate just a folder of a bucket between 2 buckets using AWS S3 replication feature?

P.S.: I don't want to replicate the entire bucket, just one folder of the bucket.

If it is possible, what configurations I need to add to filter that folder in the replication?

3 Answers

Yes. Amazon S3's Replication feature allows you to replicate objects at a prefix (say, folder) level from one S3 bucket to another within same region or across regions.

From the AWS S3 Replication documentation,

The objects that you want to replicate — You can replicate all of the objects in the source bucket or a subset. You identify a subset by providing a key name prefix, one or more object tags, or both in the configuration.

For example, if you configure a replication rule to replicate only objects with the key name prefix Tax/, Amazon S3 replicates objects with keys such as Tax/doc1 or Tax/doc2. But it doesn't replicate an object with the key Legal/doc3. If you specify both prefix and one or more tags, Amazon S3 replicates only objects having the specific key prefix and tags.

Refer to this guide on how to enable replication using AWS console. Step 4 talks about enabling replication at prefix level. The same can be done via Cloudformation and CLI as well.

Yes you can do this using the Cross-Region Replication feature. You can replicate the object either in the same region or a different one. The replicated object in the new bucket will keep their original storage class, object name and object permissions. However, you can change the owner to the new owner of the destination bucket.

Despite all of this, there are disadvantages of this feature:-

  • You cannot replicate objects which are present in the source bucket before you create the replication rule using CRR. Only the ones which are created after replication rule can be created.
  • You cannot use SSE-C encryption in replication.

You can do this with sync command.

aws s3 sync s3://SOURCE_BUCKET_NAME s3://NEW_BUCKET_NAME

You must grant the destination account the permissions to perform the cross-account copy.

Related