MongoDB aggregation lookup with pagination is working slow in huge amount of data

user_4706475 0
Viewed 2680

I've a collection with more than 150 000 documents in MongoDB. I'm using Mongoose ODM v5.4.2 for MongoDB in Node.js. At the time of data retrieving I'm using Aggregation lookup with $skip and $limit for pagination. My code is working fine but after 100k documents It's taking 10-15 seconds to retrieve data. But I'm showing only 100 records at a time with the help of $skip and $limit. I've already created index for foreignField. But still it's getting slow.

campaignTransactionsModel.aggregate([{
                    $match: {
                campaignId: new importModule.objectId(campaignData._id)
            }
                },
                {
                    $lookup: {
                        from: userDB,
                        localField: "userId",
                        foreignField: "_id",
                        as: "user"
                    },
                },
                {
                    $lookup: {
                        from: 'campaignterminalmodels',
                        localField: "terminalId",
                        foreignField: "_id",
                        as: "terminal"
                    },
                },
                {
                    '$facet': {
                        edges: [{
                                $sort: {
                                    [sortBy]: order
                                }
                            },
                            { $skip: skipValue },
                            { $limit: viewBy },
                        ]
                    }
                }
            ]).allowDiskUse(true).exec(function(err, docs) {
                console.log(docs);
            });
mongoose node.js mongodb
1 Answers

The query is taking longer because the server scans from beginning of input results(before skip stage) to skip the given number of docs and set the new result.

From official MongoDB docs :

The cursor.skip() method requires the server to scan from the beginning of the input results set before beginning to return results. As the offset increases, cursor.skip() will become slower.

You can use range queries to simulate similar result as of .skip() or skip stage(aggregation)

Using Range Queries

Range queries can use indexes to avoid scanning unwanted documents, typically yielding better performance as the offset grows compared to using cursor.skip() for pagination.

Descending Order

Use this procedure to implement pagination with range queries:

  • Choose a field such as _id which generally changes in a consistent direction over time and has a unique index to prevent duplicate values
  • Query for documents whose field is less than the start value using the $lt and cursor.sort() operators, and
  • Store the last-seen field value for the next query.

Increasing Order - Query for documents whose field is less than the start value using the $gt and cursor.sort() operators, and

Lets say the last doc you got has _id : objectid1, then you can query the docs who have _id : {$lt : objectid1} to get the docs in decreasing order. and for incresing order you can query the docs who have _id : {$gt : objectid1}

Read official docs on Range queries for more information.

user_50840000
Related