I have 3 Kafka brokers running. All my topics have replication factor set to 2. When I shut down one of the brokers, Kafka cluster, to my understanding, should keep working, but for some reason it doesn't.
For example, I can't run the following script, which works normally when all brokers are alive.
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group myGroup --describe
Note: This will not show information about old Zookeeper-based consumers.
Error: Executing consumer group command failed due to Failed to get offsets by times in 305000ms
The system topic that contains the offsets (__consumer_offsets) is also properly replicated. Broker 1 is down, but there's still at least one healthy replica.
./kafka-topics.sh --zookeeper localhost --topic __consumer_offsets --describe
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:2 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 3 Replicas: 3,1 Isr: 3
Topic: __consumer_offsets Partition: 1 Leader: 2 Replicas: 1,2 Isr: 2
Topic: __consumer_offsets Partition: 2 Leader: 2 Replicas: 2,3 Isr: 3,2
Topic: __consumer_offsets Partition: 3 Leader: 3 Replicas: 3,2 Isr: 2,3
Topic: __consumer_offsets Partition: 4 Leader: 3 Replicas: 1,3 Isr: 3
Topic: __consumer_offsets Partition: 5 Leader: 2 Replicas: 2,1 Isr: 2
Topic: __consumer_offsets Partition: 6 Leader: 3 Replicas: 3,1 Isr: 3
Topic: __consumer_offsets Partition: 7 Leader: 2 Replicas: 1,2 Isr: 2
Topic: __consumer_offsets Partition: 8 Leader: 2 Replicas: 2,3 Isr: 3,2
(...)
Any idea why I'm getting the "Failed to get offsets" error?