MongoDB: Strange behaviour of sort with pagination leading to duplicates

user_3192021 0
Viewed 1400

Here I've just created a collection of 9 documents {id, name}. All the documents have the same value "A" of the field name.

[
  {
    "name": "A",
    "id": 1
  },
  {
    "name": "A",
    "id": 2
  },
  {
    "name": "A",
    "id": 3
  },
  {
    "name": "A",
    "id": 4
  },
  {
    "name": "A",
    "id": 5
  },
  {
    "name": "A",
    "id": 6
  },
  {
    "name": "A",
    "id": 7
  },
  {
    "name": "A",
    "id": 8
  },
  {
    "name": "A",
    "id": 9
  }
]

I'm want to paginate this collection after sorting by name (it is useless to sort by name in my case, but I do that to prouve the strange behaviour), 3 by 3. (Page size is 3).

When I execute an aggregate pipeline with $skip 0, $limit 3 (first page):

db.collection.aggregate([
  {
    "$sort": {
      "name": 1
    }
  },
  {
    "$skip": 0
  },
  {
    "$limit": 3
  }
])

The result is :

[
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "id": 1,
    "name": "A"
  },
  {
    "_id": ObjectId("5a934e000102030405000001"),
    "id": 2,
    "name": "A"
  },
  {
    "_id": ObjectId("5a934e000102030405000002"),
    "id": 3,
    "name": "A"
  }
]

And now when I want to get the following page ($skip 3, $limit 3), the result is:

[
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "id": 1,
    "name": "A"
  },
  {
    "_id": ObjectId("5a934e000102030405000002"),
    "id": 3,
    "name": "A"
  },
  {
    "_id": ObjectId("5a934e000102030405000005"),
    "id": 6,
    "name": "A"
  }
]

We remark that documents with ids 1 and 3 are extracted again. This ends up with a bad pagination (duplicates)!

How do you explain this, when sorting on non unique column leads to this strange behaviour?

To reproduce the problem https://mongoplayground.net/p/hP7CMtA3b2f

aggregation-framework mongodb duplicates sorting pagination
1 Answers

When you run: sort -> skip(3) -> limit(3)

Mongo query optimizer changes the query order to: sort + limit(6) -> skip(3)

So mongo is making a 'limit' with the value of 'limit+skip'. And sort and limit are not two different stages, but a single stage. And the reason for this query optimization is it is better to find the biggest 6 elements and sort them than to first sort all and get top 6

Hence this strange behavior. Proof explain result :

{
    "stages" : [ 
        {
            "$cursor" : {
                "query" : {},
                "queryPlanner" : {
                    "plannerVersion" : 1,
                    "namespace" : "5ceb96f75538551e7d3bcdb8_lav.test",
                    "indexFilterSet" : false,
                    "parsedQuery" : {},
                    "winningPlan" : {
                        "stage" : "COLLSCAN",
                        "direction" : "forward"
                    },
                    "rejectedPlans" : []
                }
            }
        }, 
        {
            "$sort" : {
                "sortKey" : {
                    "name" : 1
                },
                "limit" : NumberLong(6)
            }
        }, 
        {
            "$skip" : NumberLong(3)
        }
    ],
    "ok" : 1.0,
    "operationTime" : Timestamp(1560596833, 1),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1560596835, 4),
        "signature" : {
            "hash" : { "$binary" : "ouhjbA5FjqF/EE4ySVpHdvG8HaM=", "$type" : "00" },
            "keyId" : NumberLong(6691284195030859777)
        }
    }
}

Add index on the name field, the sort will turn into stable sort. And you will get desired results.

user_49036630
Related