Generic way to get prev/next search results by id in Elasticsearch

Question

Say I have a million (many) documents in my index. I execute a search query sorting the items by some key X.

Now I have a very long list of results: [..., id1, id2, id3, ...]

Question: how do I get id1 and id3 if I know id2 but don't want to execute the whole search/don't want to get all ids?

I'm looking of a generic solution that works for any search query. Given an id that for certain exists in the results of a query, how to get prev/next by that id. The query should NOT have prior knowledge of anything else than the id whose prev/next are searched for. (In other words, if ordered by title and searched for prev/next of id X, the title of X is not known at query time, only X's id.)

It is of course possible to execute multiple search queries and achieve the same end result by getting id2 and then playing with ordering to get ids 1 and 3.

EDIT: I think Luc E's answer isn't what I'm looking for. In that scenario, knowledge of the original objects title is required to query for prev/next. I'm looking for a solution where only the id is known at query time.

Example data looks like this:

[...
{id: 32, title: 'AAA'},
{id: 12, title: 'BBB'},
{id: 99, title: 'CCC'},
{id: 3, title: 'DDD'},
{id: 1001, title: 'EEE'},
...]

What I know: id 99. What I don't know: what is title of id 99. What I want: ids of the prev/next items sorted by title field (=3 and 12).

To put it yet another way: I have id 99 but not the title in my hand. I want a query that gives me ids 3 and 12 (they are prev/next sorted by title).

user_10805757 · Answer

What you want to do is called deep scrolling, you have only two ways to make it :

The easiest way is the search_after but you will need to make two requests :

one request for id3
Another one for id1

So, in this example I am looking for id2 : 128. I can sort documents with the field title and I have get beforehand the value of title for id2 which is title_of_128.

To perform the search_after, I have to add the _id on a sub sort condition

Here is my query :

POST test/_search
{
  "size": 2,
  "search_after": ["title_of_128","128"],
  "sort": [
    {
      "title": {
        "order": "asc"
      },
      "_id": {
        "order": "asc"
      }
    }
  ]
}

The result of this query is id2 and id3

Now I inverse the direction of the sort in order to retrieve the id1 :

POST test/_search
{
  "size": 2,
  "search_after": ["title_of_128","128"],
  "sort": [
    {
      "title": {
        "order": "desc"
      },
      "_id": {
        "order": "desc"
      }
    }
  ]
}

The result of this query is id2 and id1

Note that sort with _id is deprecated and the best practice is to copy the _id in another field if you want to use search_after

Generic way to get prev/next search results by id in Elasticsearch

1 Answers