How can I see lag in a Kafka MirrorMaker 2.0 instance?

user_389899 0
Viewed 2466

I migrated from the original Kafka MirrorMaker to MirrorMaker 2.0 for replicating topics from one cluster to another. I am running a dedicated MirrorMaker cluster as described in the docs.

Let's say that I am replicating a topic named test-topic.

Cluster A       Cluster B
----------      ----------
test-topic ---> A.test-topic

How can I determine how far A.test-topic is behind test-topic?

The original MirrorMaker created consumer groups, so I referred to the lag on that consumer group. MirrorMaker 2.0 does not create a consumer group, so I cannot use that to determine the lag.

apache-kafka-mirrormaker apache-kafka-connect apache-kafka
3 Answers

I also had the same use case. In MM2, they are consuming the source cluster using consumer.assign() instead of consumer.subscribe()

Since assign doesn't need any groupId, we cannot able to get the lag using the consumer group.

As a workaround, I did the following thing:

  1. A scheduler will be run periodically say for every 15 mins.
  2. It will fetch the log end offsets of the source cluster topics and target cluster topics.
  3. We can compare these both and based on the lag, we can configure the alert.

For finding the log end offsets, we can create a consumer for the topic and can use consumer.seekToEnd and get that position. Also, you need to make sure that internal topics has to be skipped in this flow.

user_44165090
user_56068000

You can't get lags directly .MM2 creates internal topic in destination cluster {mm2-offets}.source-cluster-name.internal .MM2 commits offsets here. Message key is {topic}-{partition} and value is offset . Oneway to do it by checking the log end offset of source topic and compare it with offset committed for that partition in internal topic.

user_45866190
Related