I need to periodically backup a subset of a mongo database from production and restore it into a development database in order to diagnose issues for specific customers. Doing a full backup/restore isn't practical given the size of the database.
There are a few dozen mongoose models involved, and each typically has several references to other models via fields of type Schema.ObjectId, and my implementation works in most cases, however when the size of the backup exceeds something like 100k records I run into out of memory situations or database timeouts on the restore.
My algorithm uses Model.insertMany(docs) within an async loop inserting a few hundred documents at a time to one collection at a time, however when there are hundreds of thousands of docs involved this process inevitably consumes all memory or times out the dbase connection. Process and Db mmory is maxed and I've tried introducing timeouts in the algorithm to facility GC and experimented with the batch size (ranging from 1000 at a time to 1 at a time), but the result is invariably failure on a very large dataset.
If I use Model.connection.create(docs) instead of Model.insertMany(docs) the restore completes reliably, even with a huge dataset, but the ObjectID references in my backup are imported as strings rather than ObjectID's, and the result isn't queryable via mongoose.
I know that bulk insert is a difficult scenario and that Mongoose is doing validation etc on each document inserted, but for a backup/restore scenario validation is not required given that the target db is always a subset of the src db, and I'm wondering if there is a Model or Schema method or other technique I can use to transform a source doc into a Mongoose doc?
I could obviously write model specific methods to do this transformation but its also obviously something that mongoose already does, and I'm wondering if that transformation is exposed by any api? It would be a nice middle ground technique for this sort of bulk insert scenario.