Kafka Mirror Maker failing to replicate __consumer_offset topic

user_5195090 0
Viewed 2146

I am trying to make use of mirror maker to replicate __consumer_offsets topic along with other topics.

It is giving error as mentioned below.

[2018-10-24 16:16:03,802] ERROR Error when sending message to topic __consumer_offsets with key: 16 bytes, value: 445 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic. [2018-10-24 16:16:03,828] ERROR Error when sending message to topic __consumer_offsets with key: 29 bytes, value: 754 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic.

Is there any way to resolve this?

In one of the confluent presentation in SlideShare on slide21, it mentioned about replicating offset topic in multiple datacenters. Can someone tell me the possible way to achieve the same?

Or there is any other better way for backup and restore policy for Kafka.

consumer confluent-platform apache-kafka mirroring
1 Answers

Add this in your consumer.config:

exclude.internal.topics=false

And add this in your producer.config:

client.id=__admin_client


Reason being, Producers cannot write to internal topics unless you declare client.id=__admin_client which is used by AdminClient.scala. Found here: https://issues.apache.org/jira/browse/KAFKA-6524

user_70652450
Related