What actually norms store in Elasticsearch

Admin 1
Viewed 1899

I came across a mapping where, on some fields, which uses custom analyzer, norms are disabled.

Then I read about Norms and https://www.elastic.co/guide/en/elasticsearch/reference/current/norms.html found this official doc, but it doesn't explain clearly what exactly it stores and how actually its useful in scoring.

Below is the snippet from above link:

Norms store various normalization factors that are later used at query time in order to compute the score of a document relatively to a query.

I found some other docs which gave some more information and advised to Disable Norms for Analyzed Fields like numbers to represent the relative field length and the index time boost setting. But still I am unable to understand it completely.

So, In short I have below doubts:

  1. What exactly norms store?
  2. What exactly is relative field length and how it's useful for scoring?
  3. Default value of norms?
  4. Can I see the content of norms using some ES query?
elasticsearch
1 Answers

here is ma attempt of answer :)

What exactly norms store and What exactly is relative field length and how it's useful for scoring?

it stores information that allows elastic to know the relative field length. Why ?

How long is the field? The shorter the field, the higher the weight. If a term appears in a short field, such as a title field, it is more likely that the content of that field is about the term than if the same term appears in a much bigger body field

Default value of norms?

Norms are activated on text field and disabled on other fields.

Can I see the content of norms using some ES query?

No, norms are stored in the segment's data. But you can see the impact of the norms if you use the explain flag in your request. Somewhere in the score explanation mess you will see some thing like that : 

{
    "value": 1.4506965,
    "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
    "details": [
        {
            "value": 3,
            "description": "termFreq=3.0",
            "details": []
        },
        {
            "value": 1.2,
            "description": "parameter k1",
            "details": []
        },
        {
            "value": 0.75,
            "description": "parameter b",
            "details": []
        },
        {
            "value": 34.572754,
            "description": "avgFieldLength",
            "details": []
        },
        {
            "value": 48,
            "description": "fieldLength",
            "details": []
        }
    ]
}

where fieldLength and avgFieldLength are computed thanks to the norms data

This answer is primary based on https://www.elastic.co/fr/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables and https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html#field-norm

user_88814200
Related