Is ElasticSearch(lucene) segment a must while writing a doc onto disk?

user_8460545 0
Viewed 36

I'm wondering if a Lucene segment is required while writing a doc to disk. Here is the brief procedure how a doc goes into disk from ElasticSearch Firstly, an ElasticSearch document is written into in-memory cache (refer to LSM-tree memTable) Secondly, a bunch of documents are refreshed to page cache (every 1s) as an immutable Lucene segment (refer to LSM-tree immutable memTable) Eventually, that segment will be "fsync" onto a disk.

It seems that we can totally bypass the "page cache" part, that's to say that at the first step, when we want to create a doc in the in-memory buffer, elasticsearch creates a segment for it. Then if we want to update that doc, elasticSearch creates another segment for the updated doc. Now we have 2 segments in total. After 1 second these two segments will be merged and refreshed on the disk.

Do you know what advantage of Lucene segment is here?

elasticsearch lucene
1 Answers

The page cache is designed to minimize disk I/O by storing data in physical memory that would otherwise require disk access.
Caches supports different strategies generally, no-write, write-througn cache and write-back. Page cache is a kind of write-back strategy. Pages are flushed into disk periodically.
This mechanism accelerate data accessand reduce the number of I/O.
Lucene writes page cache to improve the system I/O throughput.
Of course page cache cost more memory and may lose data in cache.

user_66441230
Related