can we fetch more than 10000 records from open distro elasticsearch through java ?
I tried with normal basic sql query :
select id, name from employee order by id desc limit 10, 10;
if I try to fetch data with the above query it works fine but it's range should be under 10k. If I change limit (from, size) ie limit 10000, 10 then it does not return any value.
Yes, you can, the Default value of index.max_result_window which controls this limit is 10K, hence you are not getting the result, please read the desc of this setting in official ES docs
The maximum value of from + size for searches to this index. Defaults to 10000. Search requests take heap memory and time proportional to from + size and this limits that memory. See Scroll or Search After for a more efficient alternative to raising this.
When you give from and size as 10k,10, it crosses the limit of from+size of 10k. but the good news is that its a dynamic setting and can be changed without restarting the ES nodes:
PUT http://{{esHost}}:{{port}}/{{index}}/_settings
{
"index.max_result_window" : 20000
}
Check if it's updated in the index settings
GET http://{{eshost}}:{{port}}/{{index}}/_settings
{
"64142048": {
"settings": {
"index": {
"number_of_shards": "1",
"provided_name": "64142048",
"max_result_window": "20000", --> note the change
"creation_date": "1601531937379",
"number_of_replicas": "1",
"uuid": "TyEBaknxRJy9NGgma6jOmA",
"version": {
"created": "7080099"
}
}
}
}
}