What is the correct setup for ElasticSearch 7.6.2 highlighting with FVH?

user_1133649 0
Viewed 1207

How to properly setup highlighting search words in huge documents using fast vector highlighter?

I've tried documentation and the following settings for the index (as Python literal, commented alternative settings, which I also tried, with store and without):

    {
        "settings": {
            "number_of_shards": 1,
            "number_of_replicas": 0
        },
        "mappings": {
            "members": {
                "dynamic": "strict",
                "properties": {
                    "url": {
                        "type": "text",
                        "term_vector": "with_positions_offsets",
                        #"index_options": "offsets",
                        "store": True
                    },
                    "title": {
                        "type": "text",
                        #"index_options": "offsets",
                        "term_vector": "with_positions_offsets",
                        "store": True
                    },
                    "content": {
                        "type": "text",
                        #"index_options": "offsets",
                        "term_vector": "with_positions_offsets",
                        "store": True
                    }
                }
            }
        }
    }

Search done by the following query (again, commented places were tried one by one, in some combinations):

   {
        "query": {
            "multi_match": {
                "query": term,
                "fields": ["url", "title", "content"]
            },
        },
        "_source": {
            #"includes": ["url", "title", "_id"],
            # "excludes": ["content"]
        },
        "highlight": {
            "number_of_fragments": 40,
            "fragment_size": 80,
            "fields": {
                "content": {"matched_fields": ["content"]},
                #"content": {"type": "fvh", "matched_fields": ["content"]},
                #"title": {"type": "fvh", "matched_fields": ["title"]},
            }
        }
    }

The problem is, that when FVH is not used, ElasticSearch complains that "content" field is too large. (And I do not want to increase the allowed size). When I add "fvh" type, ES complain that terms vectors are needed: Even though I've checked those are there by querying document info (offsets, starts, etc):

the field [content] should be indexed with term vector with position offsets to be used with fast vector highlighter

It seems like:

  1. When I omit "type": "fvh", it is not used even though documentation mentions it's the default when "term_vector": "with_positions_offsets".
  2. I can see term vectors in the index, but ES does not find them. (indirectly, when indexing with term vectors the index is almost twice as large)
  3. All the trials included removing old index and adding it again.

It's also so treacherous, that it fails only when a large document is encountered. Highlights are there for queries, where documents are small.

What is the proper way to setup highlights in ElasticSearch 7, free edition (I tried under Ubuntu with binary deb from the vendor)?

elasticsearch highlight
1 Answers

The fvh highlighter uses the Lucene Fast Vector highlighter. This highlighter can be used on fields with term_vector set to with_positions_offsets in the mapping. The fast vector highlighter requires setting term_vector to with_positions_offsets which increases the size of the index.

you can define a mapping like below for your field.

"mappings": {
    "properties": {
      "text": {
        "type":        "text",
        "term_vector": "with_positions_offsets"
      }
    }
  }

while querying for highlight fields, you need to use "type" : "fvh"

The fast vector highlighter will be used by default for the text field because term vectors are enabled.

user_36360710
Related