RocksDB: Too many SST files of very small size

user_2149673 0
Viewed 4049

I am using RocksDBJava after running the service for sometime I can see the "Too many open files" exception. Digging through the previous issue mentioned on the portal I found out that it is because of system limitation of opening a maximum number of files. When I check the directory, which the rocksDB is using, I noticed there is over 100K sst files of 1KB size, and this can be the reason of the given error. I wanted to know is there any way by which we can configure the rocksDB to generate SST files of large size, so that the total number of files created is minimum and we can avoid this error.

Also in my current project there are many read threads and one write thread, and I open and close the connection (using RocksDB.open() and RocksDB.close()) before reading or writing to the rocksDB.

rocksdb java
3 Answers

You can also try setting a higher value for write_buffer_size that should create fewer and larger SST files.

user_161754420

By default, when you call RocksDB.open() the RocksDB reads all WAL to recover the memtable, then do a flush() writing the memtable as an SST file. That's why you got so many small SST files.

To avoid this behaviour set option avoid_flush_during_recovery to true when openning DB. The Java version is named avoidFlushDuringRecovery(). Also, never call flush or do implicitly flush (such as createCheckpoint etc.) in your code.

See the code here.

user_57398820
Related