clone s3 bucket without triggering events

user_457392 0
Viewed 37

My AWS app includes an s3 event which does some processing when an object lands in a specific bucket. I have a test environment which is a complete copy of the production system so obviously the test bucket also contains the same event processing.

Periodically I want to copy all the data from the production environment into the test environment. However if I use aws s3 sync this runs the events on every item that is added which is not what I want in this case.

Is there any way to clone a bucket without triggering the events on every single item?

amazon-s3
1 Answers

A week or so later I haven't found a solution but I've come to realize that this architecture is a bit of an anti-pattern.

Generally we like to structure an application into layers where each layer provides services to the layer above (say OS at the bottom, then data stores, then ORM, then business logic then UI). In this architecture the dependencies should only be downwards, but in my scenario I have a low-level component (the bucket) that is dependent on a higher level component (the business rule implemented by the event)

A better architecture would be to introduce something like an SQS queue, and the event trigger only writes events into the queue. In this case the queue and the lambda trigger that populates it would be considered low level components (i.e. the bucket depends on them). The high-level business logic also subscribes to the queue (i.e. depends on it)

This doesn't directly solve the problem, but it does decouple the bucket from the business logic. If I want to clone the system I can pause the queue, do the clone, empty the queue, then restart it, and I'm back where I want to be.

user_4573920
Related