Flink 1.11.1 off-heap Memory growing using RocksDbStateBackend

user_14380091 0
Viewed 569

We are currently migrating from Flink 1.9 to Flink 1.11.1 and we are seeing an unexpected off heap memory consumption. Our state is around 2TB and the used state backend is RocksDb. To do a safe migration we choose to set : state.backend.rocksdb.memory.managed to true, as it is the default value starting from flink 1.10.

In production we are seeing that the task manager memory consumption is always going up and going way higher when we ask for a savepoint without cancel. Please find here the screen of what we are seeing on one task manager : memory usage task manager screenshot

Please find here the configuration that has been set :

taskmanager.memory.task.heap.size: 30720m
taskmanager.memory.managed.size: 35840m
taskmanager.memory.task.off-heap.size: 20480m
taskmanager.memory.network.max: 5120m

Any advices on how we should ensure the capping of the memory usage ? Am i following the wrong track assuming that the problem is linked to RocksDbStateBackend ?

FYI. the 2 versions are running in parallel and only Flink 1.11.1 is acting this way.

Thanks

apache-flink
1 Answers

I faced a similar issue and resolved it by setting:

state.backend.rocksdb.memory.managed = false

This is a known memory fragmentation issue in Flink 1.10.x and 1.11.x. JIRA Link.

I had a long thread in Flink mailing list discussing the same.

user_99097530
Related