Delete s3 files after 30 days of being tagged

user_10007423 0
Viewed 367

So, with my application, users can "soft delete" uploaded files. What this does is update the Mongo document with a deletedAt and expireAt value. Files can be restored, and these will be removed from the mongo document. What I need is a way to tag objects on S3 for deletion after 30 days from the date they were tagged. At first, I was tagging each object and had a Lifecycle rule setup.. however I now understand that they only run after the Creation Date of an object.

What is the best (and most efficient) way to setup this system with my S3 bucket? Is it really invoking a Lambda function each day and listing all objects with a specific tag and checking the date? Could potentially have tens of millions of objects.

node.js amazon-web-services amazon-s3
1 Answers

There are a few things hidden in the lifecycle rules documentation that give you ways of dealing with this.

Firstly, the expiration time is actually based on the last modified timestamp of the object:

Amazon S3 maintains only the last modified date for each object. For example, the Amazon S3 console shows the Last Modified date in the object Properties pane. When you initially create a new object, this date reflects the date the object is created. If you replace the object, the date changes accordingly. So when we use the term creation date, it is synonymous with the term last modified date.

So if you make sure the "last modified" timestamp changes at the same time you add the tag, and not afterwards, you could use an expiration of 30 Days.

More reliably, you can create rules which specify a date rather than an age:

When specifying an action in an S3 Lifecycle rule, you can specify a date when you want Amazon S3 to take the action. When the specific date arrives, Amazon S3 applies the action to all qualified objects (based on the filter criteria).

None of the examples seems to show this, and the documentation just says:

The date value must conform to the ISO 8601 format. The time is always midnight UTC.

So I think you would need to programmatically create rules that looked something like this:

<LifecycleConfiguration>
  <Rule>
    <ID>Expire-2022-01-01</ID>
    <Tag>
       <Key>expiry</Key>
       <Value>2022-01-01</Value>
    </Tag>
    <Status>Enabled</Status>
    <Expiration>
      <Date>2022-01-01</Date>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

If I have understood the documentation correctly, this example means any object with an "expiry" tag of "2022-01-01" should be deleted at midnight UTC on 1st January 2022.

If you create a list of these for different dates, you can mark objects to be deleted at specific points in the future by adding the appropriate tag. Exactly how many rules you create depends how much it matters if the file stays around for a few extra days.

user_1579570
Related