I'm in the midst of migrating a kafka cluster (1.0.0) to a new kafka cluster (3.1). I'm using MirrorMaker2 to mirror the source cluster to the target cluster. My MirrorMaker2 setup looks something like
refresh_groups_interval_seconds = 60
refresh_topics_enabled = true
refresh_topics_interval_seconds = 60
sync_group_offsets_enabled = true
sync_topic_configs_enabled = true
emit_checkpoints_enabled = true
When looking at topics which doesn't have any migrated consumer groups, everything looks fine. When I migrate a consumer group to consumer from the target cluster (Kafka 3.1), some consumer groups are migrated successfully, while some get a huge negative lag on some partitions. This results in a lot of
Reader-18: ignoring already consumed offset <message_offset> for <topic>-<partition>
At first I didn't think of this as a big problem, I just figured that it would eventually get caught up, but after some investigation, this is a problem. I produced a new message on the source cluster, checked which offset and partition that specific message landed on the target cluster, and noticed that the migrated consumer decided to ignore that new message and log
Reader-18: ignoring already consumed offset <message_offset> for <topic>-<partition>
After that I found https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/src/main/java/com/google/cloud/teleport/kafka/connector/KafkaUnboundedReader.java#L202 So for some reason my consumer thinks its offset is much lower than it should be - on some partitions, not all. Any ideas on what can be wrong? It should also be mentioned that the offset difference on the different partitions can be quite huge, almost stretching to an order of magnitude in difference.
p.s when migrating I noticed that I'm unable to do update a job. I have to kill the job and start a new one.