I have a REST API that serves a list of resources from my MongoDB. Because I need to serve many resources, the REST API serves them paginated.

My problem is that while the client iterates over the resources, more resources might be added. In this scenario, the client might get an inaccurate list (some elements might show more than once, & some elements might not show at all).

For example, if the resources are people, and they are sorted by name. the pagination requests will returen:

The simple case where everything works as expected

first request

[
    {"name": "Alice"},
    {"name": "Charles"}
]

second request

[
    {"name": "David"},
    {"name": "Elvis"}
]

The edge case when unexpected behavior happends

But if after the first request is completed and before the second request starts someone else adds the person Bob, the users request will be:

First request

[
    {"name": "Alice"},
    {"name": "Charles"}
]

Now someone added Bob
Second Request

[
    {"name": "Charles"},
    {"name": "David"}
]

Third Request

[
    {"name": "Elvis"}
]

Note that in this case the client got Charles twice, and didn't get Bob at all.

Do you have an idea on how this can be solved?

Possible Solutions

Collection version

The solution is to store some sorts of version value for the collection, & every time the collection changes the version will be updated. Then if the clients paginates over the list & notices that the collection version changed, then they can start the pagination over again.

Problems

• I have to maintain the version field & update it every time I add records to the collection.
• The version applies to the entire collection. If the users queries only a subset of the collections (for example all people with age over 18), there is no version specific to that subset.

Resource list hash

The solution is to calculate and return the hash of the entire list (all pages) before serving the paginated result. Then if the client sees that the hash changed while iterating over the pages, than the client can start the iteration over again.
The advantage of this method over the previous one is that we don't have to maintain a version field every time we write to the database, & the hash is specific to any subset of the collection.

Problems

• Every time we retrieve a single page from the database, we must re-calculate the hash of the entire collection. this is a heavy operation.

Non of the solutions mentioned above are good enough for me. do you have any better ideas?