Elasticsearch Search Scroll API doesn't retrieve all the documents from an index

user_10627225 0
Viewed 1722

I would like to retrieve all the documents from Elasticsearch, so I referred to the Search Scroll API.

But my question is, it is not returning all the documents, I have 36 documents in one index, for that it was returning 26 only.

Even when I checked with another index, where I have more than 10k documents, there it is also not returning the last 10 documents.

I really don't know why it was returning it like that! Any help will be appreciated! Thanks in advance!

Below the code I've tried:

final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
SearchRequest searchRequest = new SearchRequest("myindex");
searchRequest.scroll(scroll);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query("")//here some query;
searchRequest.source(searchSourceBuilder);

SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT); 
String scrollId = searchResponse.getScrollId();
SearchHit[] searchHits = searchResponse.getHits().getHits();

while (searchHits != null && searchHits.length > 0) { 
    
    SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId); 
    scrollRequest.scroll(scroll);
    searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
    scrollId = searchResponse.getScrollId();
    searchHits = searchResponse.getHits().getHits();
    for (SearchHits hit: searchHits){
       String source=hit.getSourceAsString();
    }
}

ClearScrollRequest clearScrollRequest = new ClearScrollRequest(); 
clearScrollRequest.addScrollId(scrollId);
ClearScrollResponse clearScrollResponse = client.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
boolean succeeded = clearScrollResponse.isSucceeded();
elasticsearch-java-api resthighlevelclient elasticsearch
2 Answers

Today I faced with the same problem while working with an example from:

Elastic Scroll API

First of all, about documents you missed - 10 is default value for the size of requests and based on this we can suppose that one of your requests wasn't handled properly. In your code first batch of 10 documents isn't handled:

SearchHit[] searchHits = searchResponse.getHits().getHits();

Before while loop you should iterate over your searchHits . From the first time it was not clear to me in the official documents.

user_105769660

You should change your while loop logic to execute the hit iteration first and the scroll after.

while (searchHits != null && searchHits.length > 0) {

    // execute this block first otherwise the scroll will overwrite the initial hits.
    for (SearchHits hit: searchHits){
        String source=hit.getSourceAsString();
    }

    SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
    scrollRequest.scroll(scroll);
    searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
    scrollId = searchResponse.getScrollId();
    searchHits = searchResponse.getHits().getHits();
}

Another thing to consider is that you can increase the response hit size. from the docs:

The index.max_result_window which defaults to 10,000 is a safeguard, search requests take heap memory and time proportional to from + size.

So the defaulted value for max_result_window is 10k hits, you can also set this value to be something else. this means you can fetch up to 10k hits in 1 search call instead of executing redundant paginations.

You can do this by specifying the size property for searchSourceBuilder before executing the search call like so:

searchSourceBuilder.size(10000);
user_107703700
Related