Elasticsearch Spring boot findAll Result window is too large, from + size must be less than or equal to: [10000] but was [331576]

user_5327707 0
Viewed 1439

I am new to elasticsearch and im using spring data elasticsearch as such (https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#reference).

This is my pom.xml

<properties>
        <spring-data-elasticsearch.version>3.2.6</spring-data-elasticsearch.version>

...
<!-- Elasticsearch -->
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-elasticsearch</artifactId>
            <version>${spring-data-elasticsearch.version}.RELEASE</version>
        </dependency>
    </properties>

I have the code

Iterable<Data> dataList = this.dataRepository.findAll();

My DataRepository is as such

public interface DataRepository extends ElasticsearchRepository<myData, String> {
}

I am getting the error

"Result window is too large, from + size must be less than or equal to: [10000] but was [331576]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"dfs","grouped":true,"failed_shards":[{"shard":0,"index":"hardwarezone_index","node":"Psv3GnjpQ52aNB52QfvWWw","reason":{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [331576]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}]},"status":500}

I need to retrieve all the data to do some processing on it. How can i do this?

spring-data-jpa elasticsearch spring-boot
1 Answers

This is a elasticsearch limitation (and for good reasons). You can not (and should not) load huge dataset in single search operation. Elasticsearch scroll APIs are designed precisely for this purpose.

spring-data-elasticsearch do support scroll API transparenlty. All you have to do is change the return type for your repository method from Iterable<Data> to Stream<Data> and spring-data-elasticsearch will start using scroll APIs under the hood. This is documented in section 11.2. Using Scroll For Big Result Set of spring-data-elasticsearch documentation.

On a side note, though this will reduce load on storage and persistence layer, it'll likely lead to huge memory requirement in your service and controller layer if you convert this data to a Collection instead of using stream till your controller and HttpConverter layer within spring. This, however, is a separate topic. Thought to share it with you in case you are planning to use it for a production use case.

user_26416830
Related