Elasticsearch aggregation shows incorrect total

user_1382112 0
Viewed 229

Elasticsearch version is 7.4.2

I suck at Elasticsearch and I'm trying to figure out what's wrong with this query.

{
  "size": 10,
  "from": 0,
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "firstName"
          }
        },
        {
          "query_string": {
            "query": "*",
            "fields": [
              "params.display",
              "params.description",
              "params.name",
              "lastName"
            ]
          }
        },
        {
          "match": {
            "status": "DONE"
          }
        }
      ],
      "filter": [
        {
          "term": {
            "success": true
          }
        }
      ]
    }
  },
  "sort": {
    "createDate": "desc"
  },
  "collapse": {
    "field": "lastName.keyword",
    "inner_hits": {
      "name": "lastChange",
      "size": 1,
      "sort": [
        {
          "createDate": "desc"
        }
      ]
    }
  },
  "aggs": {
    "total": {
      "cardinality": {
        "field": "lastName.keyword"
      }
    }
  }
}

It returns:

    "aggregations": {
        "total": {
            "value": 429896
        }
    }

So ~430k results, but in pagination we stop getting results around the 426k mark. Meaning, when I run the query with

{
  "size": 10,
  "from": 427000,
...
}

I get:

{
    "took": 2215,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 10000,
            "relation": "gte"
        },
        "max_score": null,
        "hits": []
    },
    "aggregations": {
        "total": {
            "value": 429896
        }
    }
}

But if I change from to be 426000 I still get results.

elasticsearch aggregation
1 Answers

You are comparing the cardinality aggregation value of your field lastName.keyword to your total documents in the index, which is two different things.

You can check the total no of documents in your index using the count API and from/size you are defined at query level ie it brings the documents matching your search query and as you don't have track_total_hits it shows 10k with relation gte means there are more than 10k documents matching your search query.

When it comes to your aggregation, I can see in both the case it returns the count as 429896 as this aggregation is not depend on the from/size you are mentioning for your query.

user_40394310
Related