Please differentiate Kafka Leader(Broker) and Partition leader. Partition leader can be present in follower as well. When we send the message to Kafka broker, it will directly send the message to leader and follower based on partition?
2 Answers
From https://kafka.apache.org/documentation/#intro_distribution
Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". The leader handles all read and write requests for the partition while the followers passively replicate the leader.
[edit]
From https://kafka.apache.org/documentation/#design_replicamanagment
It is also important to optimize the leadership election process as that is the critical window of unavailability. A naive implementation of leader election would end up running an election per partition for all partitions a node hosted when that node failed. Instead, we elect one of the brokers as the "controller". This controller detects failures at the broker level and is responsible for changing the leader of all affected partitions in a failed broker. The result is that we are able to batch together many of the required leadership change notifications which makes the election process far cheaper and faster for a large number of partitions. If the controller fails, one of the surviving brokers will become the new controller.
The broker writes messages to only partition leader of topic those messages replicated to follower partitions.
If leader partition fail follower partition will replace the leader partition.
What is kafka controller? A. The Kafka controller is brain of the Kafka cluster: Broker leader is called kafka controller.
one of the brokers serves as the active controller there can be only one controller at a time. if u find 2 controllers that is a glitch.
controller monitors the liveliness of the brokers and acts on broker failures i.e. if a broker dies, then controller divides up leadership of its topic partitions to the remaining brokers in the cluster.
also responsible for electing topic partition leaders.
zookeeper elect one of the broker as controller. broker controller will elect one of the partition as leader.
When the Zookeeper doesn't receive heartbeat messages from the Controller, Zookeeper elects one of the available brokers as new controller.
Related