why merging of segments in costly CPU-wise in Elasticsearch?

user_3536516 0
Viewed 319

I have seen that every time I have a high CPU problem with ES, it's always Lucene Merge Thread.

From what I understand, segments are already sorted, so you are just merging two sorted segments every time, à la merge sort's merge process. Why would merging be so costly. Or am I missing something?

elasticsearch lucene
1 Answers

There are couple of factors why merging process is costly on IO and CPU:

  1. Lucene uses Skip list data structure they can be costly to merge, there is a good article about Skip list merge here.
  2. Couple of parallel merging can happen at the same time.
  3. Lucene will need to create a third segment and merge both segment, so you will need enough space for it.

There is a good blog post about segment merging here.

user_140183850
Related