Mirror Maker 2 - Failback from Stanby cluster

user0e9m1 0
Viewed 12

We have 2 clusters Apache Kafka architecture - Active-Standby. Mirror Maker 2 is used to replicate all data from Active cluster to Standby one. If Active cluster goes down, we will failover to Standby cluster and work on it (connect producers, consumers and send new data).

The question is - what is a correct way to failback from Standby cluster to Active one keeping all the data that was sent to the Stanby cluster while it was active ?

Do we need to run Mirror Maker in opposite direction to replicate data from Standby cluster to Active cluster and once data is replicated failback to Active cluster ?

How can we stop producers to send new data to Standby cluster to complete replication without losing any data ?

Could you please advise ?

apache-kafka-mirrormaker apache-kafka
1 Answers

How can we stop producers to send new data to Standby cluster to complete replication without losing any data ?

From what I've seen documented by Confluent, you would do a DNS remapping to the active cluster for a single bootstrap.servers address, and not change any client code. When a cluster goes down, producers will need to be pre-configured with retries then establish new connections automatically via DNS query, and consumers will (in theory) do a periodic metadata refresh/rebalance on their own. Otherwise, if you stop a producer, you'll lose its current unflushed buffer, or stop a consumer from processing/committing its current batch.

In practice, I'm not sure how reliable that is since it'll require DNS caching TTL to be lowered/disabled in your runtime environment.

For example, in Java

java.security.Security.setProperty("networkaddress.cache.ttl", "0");

Do we need to run Mirror Maker in opposite direction to replicate data from Standby cluster to Active cluster

Yes. MirrorMaker2 has loop detection, but you will need to make your consumers subscribe to a topic pattern like my-topic(.replica)? to read both any direct produced topic (in the local cluster) and the mirrored one from the remote cluster. Ideally you also implement an ACL policy that only MirrorMaker producer clients can write to any .replica topic. Keep in mind that message ordering will be mostly lost when consuming from more than one topic; it'll only be ordered within the individual partitions of the respective topics.

OneCricketeer0
Related