Optimize async queries nodeJs

user_11982575 0
Viewed 145

I am working on a project, which has over 25k users, everytime any record is created for Oppurtinity module, it executes this function, which inserts notification details for each user in mongodb, The problem is that it uses up the cpu and doesn't let any other APIs to produce results, until this async ends it takes up time, meanwhile the server doesn't load any other data/it gives late response.

  1. Node- v10.15.3
  2. MongoDB shell version v4.0.17
  3. Server=Windows=Memory-8GB
  4. CPU 2.70GHz speed

I want to know, what is the problem here, is it the code, if so how can i optimize it?

async.eachSeries(users, (eachuser, next) => {
     db.notifications.find({user_id:eachuser._id},function(errr,opp_noti){
        if(opp_noti.length != 0){
          db.notifications.update( { user_id:eachuser._id},{$push:{oppurtunity:oppurtunity_id}},function(errr,result){
            })
        }else{
        db.notifications.insert({user_id:eachuser._id,oppurtunity:[oppurtunity_id]},function(errr,result){
         })
        }
       })
    next();
})
async-await node.js mongodb optimization
1 Answers

There were a few problems with your sample code, and your question. I had to assume you were using these libraries: Async and MongoDB. Please try to organise your questions a little better.

If I have assumed correctly, then you didn't follow the docs carefully as you haven't implemented the async.eachSeries function properly - eachSeries expects an async function as the second argument.

I have rewritten your code based on the docs for those two libraries. I have never used them before, but I'm reasonably confident it's correct.

async.eachSeries(users, async (user) => {
    const user_id = user._id;
    const cursor = db.notifications.find({ user_id });
    const count = await cursor.count();
    if (count > 0) {
        return db.notifications.update({ user_id },
            {
                $push: { oppurtunity: oppurtunity_id } // where is oppurtunity_id defined?
            }
        )
    } else {
        return db.notifications.insert({
            user_id,
            oppurtunity: [oppurtunity_id]
        })
    }
})

You probably already realise this, but async.eachSeries will throttle the requests by running them in series. While this may be necessary for various reasons, it will slow down the process and doesn't take advantage of Node's non-blocking nature. async.each will run them all in parallel, which in theory is fastest, but at 25K records it could cause problems. So I suggest you take the middle ground and consider using async.eachOfLimit, setting a parallel limit and seeing how it performs.

You can also break down the problem further and get a big performance increase by not querying the DB for each user. Try getting all the users that need updating in advance in one query.

Something like the following will return all ID's in the notifications collection in one query (probably more than one under the hood, but still much better than n).

const user_ids = db.notifications.find({}, { _id:1 }).map(item => item._id)

Once you have the ID's that exist, you can create two workloads, one to run all your db.notifications.update queries, and another to run all your db.notifications.insert queries.

user_46326270
Related