Find all documents in one collection, change data and save into another collection using mongoose and node express

user_5739642 0
Viewed 41

I'm trying get select data from 1 collection, remove the fields I don't need, and add some new fields, and passing the data needed which is being replicated on each document for the new fields, and save them into a different collection.

Am I getting close or am I way off? Any help would be appreciated.

router.post('', (req, res, next) => {
    const taskDay = req.body.taskDay;
    switch (taskDay) {
        case 'taskSunday':
            MasterTask.find({taskSunday: true}).forEach(tempData => {
                const taskSheet = new Tasksheet({
                    taskDate: req.body.taskDate,
                    taskTicket: req.body.taskTicket,
                    taskTime: tempData.taskTime,
                    taskCompany: tempData.taskCompany,
                    taskDescription: tempData.taskDescription,
                    taskCompleted: req.body.taskCompleted,
                    taskComments: req.body.taskComments,
                    taskCompletedBy: req.body.taskCompletedBy,
                    taskTimeStamp: req.body.taskTimeStamp,
                    taskKnowledgeTitle: tempData.taskKnowledgeTitle,
                    taskKnowledgeUrl: tempData.taskKnowledgeUrl,
                    taskType: tempData.taskType
                }); 
                Tasksheet.insertMany(taskSheet).then(result => {
                    console.log(result);
                    res.status(200).json({message: 'task sheet created'})
                });
            });
            break;

Sample Data being pulled in find:

    _id: "5f0acb7209e5981d10bb765a",
    taskTime: "00:00",
    taskCompany: "CompanyName",
    taskDescription: "Some Needed Daily Task",
    taskSunday: true,
    taskMonday: true,
    taskTuesday: true,
    taskWednesday: true,
    taskThursday: true,
    taskFriday: true,
    taskSaturday: true,
    taskDisable: false,
    taskEndOfMonth: null,
    taskKnowledgeTitle: "Some Title",
    taskKnowledgeUrl: "Some URL",
    taskForSpecificDays: null,
    taskType: "Core",
    __v: 0
  }

Data transformation that I'm trying to save to new collection:

  {
    id: "mongooseID",
    taskTicket: "",
    taskDate: null,
    taskTime: "00:00",
    taskCompany: "somename",
    taskDescription: "This is an example description of a task.  This is going to be an extra long line of text to test.",
    taskCompleted: false,
    taskComments: "",
    taskCompletedBy: "",
    taskTimeStamp: "",
    taskKnowledgeTitle: "Test",
    taskKnowledgeUrl: "http://www.google.com",
    taskType: "Core"
  }
angular mongoose express node.js mongodb
2 Answers

Try this.

MasterTask.find(
{
    taskSunday: true
},
function(err, tempData) {
    let taskSheet = [];
    tempData.forEach(t => {
        taskSheet.push({
            taskDate: req.body.taskDate,
            taskTicket: req.body.taskTicket,
            taskTime: t.taskTime,
            taskCompany: t.taskCompany,
            taskDescription: t.taskDescription,
            taskCompleted: req.body.taskCompleted,
            taskComments: req.body.taskComments,
            taskCompletedBy: req.body.taskCompletedBy,
            taskTimeStamp: req.body.taskTimeStamp,
            taskKnowledgeTitle: t.taskKnowledgeTitle,
            taskKnowledgeUrl: t.taskKnowledgeUrl,
            taskType: t.taskType
        })
    })

    Tasksheet.insertMany(taskSheet, (err, t) => {
        console.log(err)
        console.log(t)
    })
})
user_49564980

Well, you do it a bit wrong. If you want to simply re-export your collection values (without modifying it) you'd better use such utilities like mongodump or mongoexport (depends on scenario) and then import it via mongorestore or mongoimport

If you want to handle your problem with JS code, it's fine, but code above is a bit wrong, why is that? Because if you deal with real huge number of documents in a collection, this code will never be finished.

So it's better to use find query with a cursor, like this:

async function t () {
   try {
        await collection_name.find({taskSunday: true}).lean().cursor({batchSize: 10}).eachAsync(async (document) => {
                /**
                * Your could write any logic here related with your document
                */
                const taskSheet = new Tasksheet({
                    taskDate: document.taskDate,
                    taskTicket: document.taskTicket,
                    taskTime: document.taskTime,
                    taskCompany: document.taskCompany,
                    taskDescription: document.taskDescription,
                    taskCompleted: document.taskCompleted,
                    taskComments: document.taskComments,
                    taskCompletedBy: document.taskCompletedBy,
                    taskTimeStamp: document.taskTimeStamp,
                    taskKnowledgeTitle: document.taskKnowledgeTitle,
                    taskKnowledgeUrl: document.taskKnowledgeUrl,
                    taskType: document.taskType
                }); 
                taskSheet.save()
        }, { parallel: 10})
   } catch (e) {
       console.error(e)
   }
}

This code will be guaranteed finished, because it takes every 10 documents (it depends on {batchSize: 10} and { parallel: 10} values) from your original collection_name and insert in one-by-one in 10 streams. Will allows you not load all the collection in your RAM.

And also allows you to modify the necessary data on-the-fly

user_74756150
Related