elasticsearch 7.9 is throwing a new error "illegal_argument_exception" because the text is too big

user_1130176 0
Viewed 3096

After upgrading to ES 7.9, I now get this error upon searching:

{"error"=>{"root_cause"=>[{"type"=>"illegal_argument_exception", "reason"=>"The length of [data.Basic Information.Doc] field of [59921e665c3e743c5befb1c4] doc of [cases] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!"}], "type"=>"search_phase_execution_exception", "reason"=>"all shards failed", "phase"=>"query", "grouped"=>true, "failed_shards"=>[{"shard"=>0, "index"=>"cases", "node"=>"Wrz1BVCJRgOyGOFxC0otMQ", "reason"=>{"type"=>"illegal_argument_exception", "reason"=>"The length of [data.Basic Information.Doc] field of [59921e665c3e743c5befb1c4] doc of [cases] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!"}}], "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"The length of [data.Basic Information.Doc] field of [59921e665c3e743c5befb1c4] doc of [cases] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"The length of [data.Basic Information.Doc] field of [59921e665c3e743c5befb1c4] doc of [cases] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!"}}}, "status"=>400}, @response=#<Net::HTTPBadRequest 400 Bad Request readbody=true>, @headers={"content-type"=>["application/json; charset=UTF-8"], "content-length"=>["1840"]}>

My understanding is that I have to somehow set the analyzer or the index setting to use term_vectors with offsets, but I don't know how to do this.

Is this something I change in elastic.yml, or do I fire off a curl command (if so, can you help with the curl command).

thanks, kevin

elasticsearch
1 Answers

Maybe this would solve the problem.

curl -XPUT "localhost:9200/INDEX/_settings" -H 'Content-Type: application/json' -d' {
    "index" : {
        "highlight.max_analyzed_offset" : 60000000. // Set this value according to your requirement.
    }
}
'

Also remember:

Plain highlighting for large texts may require substantial amount of time and memory. To protect against this, the maximum number of text characters that will be analyzed has been limited to 1000000. This default limit can be changed for a particular index with the index setting index.highlight.max_analyzed_offset.

Go through the documentation to understand about highlighting.

user_52701860
Related