Kafka Mirror maker : Getting consumer lag in negative (-x)

user_7768701 0
Viewed 629

I am running Apache MirrorMaker 3.0.0 with active-active strategy on two Kafka clusters (named DC, DR). So topic on DC is replicated by MirrorMaker2 as DC.<topic_name> on the DR cluster and vice-vera.

Use case: Produced 20 records in DC cluster topic named: notification-test1 and consumed all 20 records from same cluster so lag is 0 and records are flushed due to retention period (i have set retention period to 10 min). After that i have started DR cluster and kafka MirrorMaker2.

Expectation:

 Current-Offset : 0 (because Initial 20 records are flushed due to retention period. With delta difference, cluster1 21th record is equivalent to 1st record of cluster2 )
 Log_end_offset : 0 (same reason)
 Lag : 0

Observation :

 Current-Offset : 20 (Not expected)
 Log_end_offset : 0 
 Lag : -20 ( 0 - 20 ) Lag is negative

screenshot:

enter image description here

Kindly suggest if i have missed anything.

Thanks in advance..!!

apache-kafka-mirrormaker apache-kafka
1 Answers

You're most likely hitting a known issue: KAFKA-12635.

This happens when MirrorMaker is started on a topic with some empty partitions due to retention. In this case, the committed offset translation fails and MirrorMaker commits offsets larger than log end offsets in the target cluster, hence you get negative lag!

Unfortunately this has not been fixed yet. I have proposed a solution in this PR.

Until a fix is released, if you produce records to the affected partitions on your source cluster, it should sort itself out eventually.

user_17651890
Related