Kafka, questions about setting up

user_17130628 0
Viewed 98

I'm testing Kafka on Linux, but I don't know what's wrong because the test results are different from what I understand.

Let me explain the setting.

Currently, three brokers were configured with kafka version 2.8.1 in centos7 using 9092, 9093, and 9094 ports, respectively. In the case of producers, all three ports were connected to the bootstrap-server setting and then executed.

kafka-console-producer.bat --broker-list localhost:9092,localhost:9093,localhost:9094 --topic test

In the case of consumers, three were set up so that they could be attached to each of the three ports.

1. kafka-console-consumer.bat --bootstrap-serverlocalhost:9092 --topic test
2. kafka-console-consumer.bat --bootstrap-serverlocalhost:9093 --topic test
3. kafka-console-consumer.bat --bootstrap-serverlocalhost:9094 --topic test

If I were to explain what I understood here,

In the case of Kafka, the leader broker acts as a controller, and the follow brokers copy the leader broker's data. If one of the follower brokers dies, a disconnection message simply appears on the consumer connected to the broker.Other brokers operate normally. If a leader broker dies, one of the follow brokers will be changed to a leader broker, and the changed leader broker will act as a controller.

If I were to explain the problem, If you kill a leader broker, check the describe option, and the other follow broker has changed to a leader, but both producers and consumers cannot find a new leader and fail. Even if a broker running on 9092 ports kills the broker without being a leader, the producer and consumer will fail.

Question.

  1. If the leader broker dies, should the producer and consumer also set up a new connection?
  2. Am I misunderstanding the producer and consumer?
  3. Is there anything wrong with the setting?
apache-kafka
1 Answers

I'm testing Kafka on Linux

But you're using Batch files, and connecting to localhost, which are for Windows...

so that they could be attached to each of the three ports.

This isn't how Kafka distributes load. You can only have one consumer thread active per topic partition. Unclear how many partitions your topic has, but if you have only one and that specific broker died (it is the only replica and leader), this explains why your clients would stop working.

Besides that, Kafka is generally on the same port, on mulitple hosts. Using one host is not truly fault-tolerant, and is a waste of resources (CPU, RAM, and disk).

Regarding producers, there is a property for retries that can be configured; I'm not sure if the console producer overrides the default or not, but it should connect to the next available broker upon a new request.

For consumers, the same, however, you'll want to make sure your offsets.topic.replication.factor (and transactions topic factor, if you use them) is higher than 1; otherwise, consumers will be unable to read anything (or transactions will not work, which are enabled by default in newer versions)

user_23086830
Related