What is the best method to migrate __consumer_offsets to a new cluster?

user_7030682 0
Viewed 204

I'm in the phase of migrating a big cluster to a new datacenter. I'm using the MirrorMaker tool to do the mirroring of the topics. I have a requirement to migrate the __consumer_offset topic to the new cluster. What is the procedure to move this topic?.

My consumer properties 

bootstrap.servers=<server_dns>:9092
exclude.internal.topics=false
client.id=mirror_maker_consumer_all
group.id=mirror_maker_consumer_all

producer.properties

bootstrap.servers=<bootstrap_servers>:9092
acks = 1
batch.size = 10000
timeout.ms = 3000
client.id=mirror_maker_consumer_offsets

I'm running mirror maker with num.streams=10 and whitelist='.*'

cloudera apache-kafka-mirrormaker apache-kafka
1 Answers

With MirrorMaker you can't mirror that topic. Offsets across clusters often differ so the data in that topic does not make sense in the new cluster.

MirrorMaker2 addresses this issue and is able to replicate offsets between clusters. MirrorMaker2 is now the recommend tool and replaces the old MirrorMaker.

See the MirrorMaker2 README for details about its features and how to run it: https://github.com/apache/kafka/tree/trunk/connect/mirror

user_17651890
Related