Setting up Kafka HA with MirrorMaker - how to handle that in Flink?

user_4396113 0
Viewed 298

We are setting up MirrorMaker 2.0 for Kafka. If I understand correctly, topic offsets are not equal in replicated cluster. This is not a problem for regular Kafka app as consumer groups get replicated too. Flink stores Kafka offsets internally in state - I assume after job restart with state, things can go awry. Is there a way to set up Flink so that we can recover from cluster failure on replicated Kafka cluster? I think we should migrate state somehow but have no experience doing that.

apache-kafka-mirrormaker flink-streaming apache-flink apache-kafka
2 Answers

I don't think that migrating the state will help in this case. I can see two ways of tackling this issue, but none of them is perfect:

  1. Technically, you could disable storing offsets on checkpoint and enable enable.auto.commit in consumer, but I think this is a little dangerous in terms of possible losing data.
  2. You can extend the FlinkKafkaConsumer and modify the open function, so that it will ignore the offsets stored in state and instead it will use fetcher to fetch the offsets. This should work fine I think, since the offsets are commited to Kafka on checkpoint if the checkpointing is enabled.
user_88990440

MirrorMaker 2 manages to sync offsets via an internal topic and uses the timestamp of the messages to account for syncing and drift in the case of failover.

user_23086830
Related