One obvious advantage of using the EmbeddedRocksDBStateBackend with Flink is that it can spill to disk when there isn't enough memory. But if I'm prepared to give it enough memory so that it never needs to use the disk, how is that different from using the HashMapStateBackend?
If I give Flink's RocksDB state backend enough memory, how does that differ from the heap-based state backend?
Viewed 289
1 Answers
These are the major differences:
- The serialized format in which the RocksDB state backend maintains state has (much) less overhead (in general) than the binary object format used on the heap. So for a given amount of memory, RocksDB can hold more state.
- The ser/de overhead in RocksDB means that that backend has significantly less throughput (on average).
- The RocksDB backend maintains its state in off-heap memory, whereas state kept on the heap is subject to GC overhead and pauses. So RocksDB may have better worst-case latency. (Once Flink supports Java 17 and its modern garbage collectors, this factor may disappear.)
- The RocksDB backend supports incremental checkpointing, which can significantly speed up both snapshots and restores (but see FLIP-151).
FWIW, some users choose to deploy with RocksDB configured to use a RAM disk as the local disk.
Related