Everyday, we receive data from an external provider and we need those data to update our collection in our DB. The provider send all the data each day, so at first, we were just deleting the current data and replacing them by the new data.
The problem is, we are listening to those changes to create an other collection and there is a lot of data every day so it takes a lot of time if each day we delete and insert.
So the idea would be to :
- Insert the new data in a temp collection
- Compare those data with the older data
- Upsert the new data in the collection, and delete the data that are not in the temp collection
I think I have to use either a pipeline for this or the updateMany/replaceOne methods, but idk if it's really possible to do so.
ReplaceOne looks like a good idea, using bulkWrite, but then I also need to delete every data that are not in the temp collection.
Any idea would be great, thanks