What will flink do when zookeeper data reset?

jimin 0
Viewed 40

Our Flink uses Zookeeper as HA service. Therefore, Zookeeper continuously keeps and stores the snapshots of Flink data. (There is also Kafka connected to Zookeeper)

I am curious about if Zookeeper nodes are all reset and all snapshot data disappeared, what Flink will do next?

flink-streaming apache-zookeeper apache-flink
1 Answers

Flink doesn't use Zookeeper for storing snapshots of Flink data. It uses Zookeeper to determine what is the active JobManager and if it should switch to another one, to achieve a high availability setup for Flink's JobManager.

Snapshots are written to a high available storage directory, like S3 or HDFS. See https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/ha/zookeeper_ha/

If the Zookeeper data is gone, it means that Flink doesn't know what's the currently active JobManager and therefore the cluster and its jobs will cancel.

Martijn Visser0
Related