Elastic Search sorting very slow on large datasets

user_1035104 0
Viewed 1063

The sorting of data in ES was very fast when I had less data, but when the data increased into GBs then the sorting of the fields is very very slow, normal fields < 1 sec, but for the fields with the below mapping the sorting time is > 10 seconds and sometimes more.

I am unable to figure out why is that? can anyone help me with this?

Mapping:

"newFields": {
    "type": "nested",
    "properties": {
      "group": { "type": "keyword" },
      "fieldType": { "type": "keyword" },
      "name": { "type": "keyword" },
      "stringValue": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256,
            "normalizer": "sort_normalizer"
          }
        }
      },
      "longValue": {
        "type": "long"
      },
      "doubleValue": {
        "type": "float"
      },
      "booleanValue": {
        "type": "boolean"
      }
    }
  }

Query:

 {
  "index": "transactions-read",
  "body": {
    "query": {
      "bool": { "filter": { "bool": { "must": [{ "match_all": {} }] } } }
    },
    "sort": [
      {
        "newFields.intValue": {
          "order": "desc",
          "nested": {
            "path": "newFields",
            "filter": { "match": { "newFields.name": "johndoe" } }
          }
        }
      }
    ]
  },
  "from": 0,
  "size": 50
}

So is there any way to make it faster? Or am I missing something here?

elasticsearch
2 Answers

Nested datatype is known for bad performance and on top of it you are using sort which is again a costly operation Please refer this great medium blog of Gojek engineering team on their perf issues with nested docs.

They suggested some optimization which includes changing the schema as well but they have not covered the infra level optimization like tunning the JVM heap size and having the favourable shards and replicas which are backbones of elasticsearch and its worth checking and tunning these infra params as well.

user_40394310

Nested sort will be slower compared to non-nested sort. As the number of nested documents in your index increases - unfortunately, sort will slow down.

user_79279750
Related