Kafka Producer not able to send messages

user_3270104 0
Viewed 12019

I am very new to Kafka.

Using Kafka 0.11

Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor')

I get the above error on sending a message for a topic

kafka-topics --zookeeper localhost:2181 --topic test --describe
Topic:test1 PartitionCount:1    ReplicationFactor:1 Configs:
Topic: test1    Partition: 0    Leader: 0   Replicas: 0 Isr: 0
apache-kafka
3 Answers

In my case, my error is also similar.

ERROR [KafkaApi-2] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

Cluster : 2 Brokers (ID=1,ID=2) with hostname-1 and hostname-2
Kafka version : 1.0.1

listeners=PLAINTEXT://:9090,SSL://:9091,PLAINTEXT_EXT://:9092,SSL_EXT://:9093,SASL_SSL://:9094,SASL_PLAINTEXT://:9095

and both broker server.properties is set to offsets.topic.replication.factor=1

but I configured my advertised hostname as hostname-1 in both broker over protocols which is being used for inter-broker communication ( and thus broker with ID=2 keep on giving above error.

advertised.listeners=PLAINTEXT://hostname-2:9090,SSL://hostname-2:9091,PLAINTEXT_EXT://<EXTERNAL_IP>:9092,SSL_EXT://<EXTERNAL_IP>:9093,SASL_SSL://hostname-1:9094,SASL_PLAINTEXT://hostname-1:9095

correction on SASL_SSL and SASL_PLAINTEXT fixed this error.

PS : SASL_PLAINTEXT is security.inter.broker.protocol in this cluster. This error seems to be related with port availability as well.

user_39825190

This means your cluster have default replication factor setting to some numbers to override this you need to edit server.properties and add replication factor parameter with your choice value 

offsets.topic.replication.factor=1

In my case i wanted to run single node kafka with single node zookeeper for that case you need to create topic with replication factor 1 otherwise you will gent an error 

mansoor@c2dkb05-usea1d:~$ ./bin/kafka-topics.sh --create --zookeeper zookeeper-svc:2181 --replication-factor 2 --partitions 2 --topic mqttDeviceEvents

Error while executing topic command : Replication factor: 2 larger than available brokers: 1.
[2020-06-18 14:39:46,533] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1.

the correct way to create topic is when you have single node kafka 

mansoor@c2dkb05-usea1d:$ ./bin/kafka-topics.sh --create --zookeeper zookeeper-svc:2181 --replication-factor 1 --partitions 2 --topic mqttDeviceEvents
Created topic mqttDeviceEvents.
user_45723960
Related