What is the best way to continuously copy data from one s3 to another s3 bucket? I understand that S3 supports event notification and send them to lambda, SNS and SQS, but I am unsure what would be the option to do? should I trigger a lambda function to receive the records from S3 and copy them into another s3? should I use SNS or SQS to do that?
2 Answers
Assuming that both your buckets are in the same region (otherwise you could use Cross-Region Replication), the process would be:
- Create an AWS Lambda function
- Configure S3 events on the bucket to trigger the Lambda function when an object is created
The Lambda function will be passed details of the bucket and object. It should then copy the object to the other bucket.
There is no need to involve Amazon SNS nor Amazon SQS.
I would recommend to use S3 Replication. You can replicate objects between different AWS Regions or within the same AWS Region.
Cross-Region replication (CRR) is used to copy objects across Amazon S3 buckets in different AWS Regions.
Same-Region replication (SRR) is used to copy objects across Amazon S3 buckets in the same AWS Region.
As described here: https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.html
You just need to go to Management -> Replication in your S3 bucket and configrue the source and destination bucket. Moreover, you can use encryption:
Why Use Replication?
Replication can help you do the following:
Replicate objects while retaining metadata — You can use replication to make copies of your objects that retain all metadata, such as the original object creation time and version IDs. This capability is important if you need to ensure that your replica is identical to the source object
Replicate objects into different storage classes — You can use replication to directly put objects into Glacier, DEEP ARCHIVE, or another storage class in the destination bucket. You can also replicate your data to the same storage class and use lifecycle policies on the destination bucket to move your objects to a colder storage class as it ages.
Maintain object copies under different ownership — Regardless of who owns the source object, you can tell Amazon S3 to change replica ownership to the AWS account that owns the destination bucket. This is referred to as the owner override option. You can use this option to restrict access to object replicas.
Replicate objects within 15 minutes — You can use S3 Replication Time Control (S3 RTC) to replicate your data in the same AWS Region or across different Regions in a predictable time frame. S3 RTC replicates 99.99 percent of new objects stored in Amazon S3 within 15 minutes (backed by a service level agreement). For more information, see Replicating Objects Using S3 Replication Time Control (S3 RTC).
When to Use CRR?
Cross-Region replication can help you do the following:
Meet compliance requirements — Although Amazon S3 stores your data across multiple geographically distant Availability Zones by default, compliance requirements might dictate that you store data at even greater distances. Cross-Region replication allows you to replicate data between distant AWS Regions to satisfy these requirements.
Minimize latency — If your customers are in two geographic locations, you can minimize latency in accessing objects by maintaining object copies in AWS Regions that are geographically closer to your users.
Increase operational efficiency — If you have compute clusters in two different AWS Regions that analyze the same set of objects, you might choose to maintain object copies in those Regions.
When to Use SRR?
Same-Region replication can help you do the following:
Aggregate logs into a single bucket — If you store logs in multiple buckets or across multiple accounts, you can easily replicate logs into a single, in-Region bucket. This allows for simpler processing of logs in a single location.
Configure live replication between production and test accounts — If you or your customers have production and test accounts that use the same data, you can replicate objects between those multiple accounts, while maintaining object metadata, by implementing SRR rules.
Abide by data sovereignty laws — You might be required to store multiple copies of your data in separate AWS accounts within a certain Region. Same-Region replication can help you automatically replicate critical data when compliance regulations don't allow the data to leave your country.
Related