A good way to expire specific documents in one collection and add them to another collection on expiry

user_9110142 0
Viewed 1792

I'm using Nodejs and MongoDB driver.

A two part question

  1. I have a collection which is called openOffers which I want to expire when it hits the time closeOfferAt. I know that MongoDB offers a TTL, expireAt and expireAfterSeconds. But, when I use these, the same TTL is applied to all the documents in a particular collection. I'm not sure if I'm doing this correctly. I want document level custom expiry. Any syntax might be very useful! Docs in openOffers
{
 "_id":"12345",
 "data": {...},
 "closeOfferAt" : "2022-02-21T23:22:34.023Z"
}
  1. I want to push these expired documents to another collection openOfferLog. This is so that I can retain the document for later analysis.

Current approach: I haven't figured a way to have a customized TTL on each doc in openOffers. But, I currently insert docs into both, openOffers and openOffersLog together, and any data change in openOffers has to be separately propagated to openOffersLog to ensure consistency. There has to be a better scalable approach I suppose.

EDIT-1:

I'm looking for some syntax logic that I can use for the above use case. If not possible with the current MongoDB driver, I'm looking for an alternative solution with NodeJS code I can experiment with. I'm new to both NodeJS and MongoDB -- so any reasoning supporting the solution would be super useful as well.

aggregation-framework mongodb-query mongodb
2 Answers

There are two ways to implement TTL indexes.

  • Delete after a certain amount of time - this is you have already implemented
  • Delete at a specific clock time - for the detailed answers you can visit the MongoDB Docs

So the second option fulfills your expectation,

  1. Just set 0 (zero) in expireAfterSeconds field while creating an index,
db.collection.createIndex({ "closeOfferAt": 1 }, { expireAfterSeconds: 0 })
  1. just set the expiration date in closeOfferAt while inserting the document, this will remove the document at a particular timestamp.
db.collection.insert({
    "_id":"12345",
    "data": {...},
    "closeOfferAt" : ISODate("2022-02-23T06:14:15.840Z")
})
user_89871280

Do not make your application logic depend on TTL indexes.

In your app you should have some scheduler that runs periodic tasks, some of them would move the finished offers to other collection and delete from the original even in bulk and no need to set a TTL index.

To keep consistency nothing better than a single source of truth, so if you can, avoid deleting and only change some status flag and timestamps.

A good use of a TTL index is to automatically clear old data after a relative long time, like one month or more. This keeps collection/indexes size in check.

user_16727990
Related