Optimizing Elastic Search Index for many updates on few fields

user_65584 0
Viewed 46

We are working on a large Elasticsearch Index (>1 bi documents) with many searchable fields evenly distributed across >100 shards on 6 nodes.

Only a few of these fields are actually changed, but they are changed very often. About 25% of total requests are changes to these fields.

We have one field that simply holds a boolean, which accounts for more than 90% of the changes done to the document.

It looks like we are taking huge performance hits re-indexing the entire documents, even though a simple boolean is changing.

Reading around I found that this might be a case where one could store the boolean value within a parent-child field, as this would effectively place it in a separate index thus not forcing a recreation of the entire document. But I also read that this comes with disadvantages, such as more heap space usage for the relation.

What would be the best way to solve this challenge?

elasticsearch
1 Answers

Yes, since Elasticsearch is internally a write-only system every update effectively creates a new document and marks the old copy as stale to be garbage-collected later.

Parent-child (aka join) or nested fields can help with this but they come with a significant performance hit of their own so your search performance probably will suffer.

Another approach is to use external fast storage like Redis (as described in this article) though it would be harder to maintain and might also turn out to be inefficient for searches, depending on your use case.

General rule of thumb here is: all use cases are different and you should carefully benchmark all feasible options.

user_12151360
Related