Leader Not Available Kafka in Console Producer

What solved it for me is to set listeners like so:

advertised.listeners = PLAINTEXT://my.public.ip:9092
listeners = PLAINTEXT://0.0.0.0:9092

This makes KAFKA broker listen to all interfaces.

I have been witnessing this same issue in the last 2 weeks while working with Kafka and have been reading this Stackoverflow's post ever since.

After 2 weeks of analysis i have deduced that in my case this happens when trying to produce messages to a topic that doesn't exist.

The outcome in my case is that Kafka sends an error message back but creates, at the same time, the topic that did not exist before. So if I try to produce any message again to that topic after this event, the error will not appear anymore as the topic as been created.

PLEASE NOTE: It could be that my particular Kafka installation was configured to automatically create the topic when the same does not exist; that should explain why in my case I can see the issue only once for every topic after resetting the topics: your configuration might be different and in that case you would keep receiving the same error over and over.

Adding this since it may help others. A Common problem can be a misconfiguration of advertised.host.name. With Docker using docker-compose setting the name of the service inside KAFKA_ADVERTISED_HOST_NAME wont work unless you set the hostname as well. docker-compose.yml example:

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    hostname: kafka
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

The above without hostname: kafka can issue a LEADER_NOT_AVAILABLE when trying to connect. You can find an example of a working docker-compose configuration here

In my case, it was working fine at home, but it was failing in office, the moment I connect to office network.

So modified the config/server.properties listeners=PLAINTEXT://:9092 to listeners=PLAINTEXT://localhost:9092

In my case, I was getting while describing the Consumer Group

If you are running kafka on local machine, try updating $KAFKA_DIR/config/server.properties with below line: listeners=PLAINTEXT://localhost:9092 and then restarting kafka.

Since I wanted my kafka broker to connect with remote producers and consumers, So I don't want advertised.listener to be commented out. In my case, (running kafka on kubernetes), I found out that my kafka pod was not assigned any Cluster IP. By removing the line clusterIP: None from services.yml, the kubernetes assigns an internal-ip to kafka pod. This resolved my issue of LEADER_NOT_AVAILABLE and also remote connection of kafka producers/consumers.

If you get repeated error messages like this:

Error while fetching metadata with correlation id 3991 : {your.topic=LEADER_NOT_AVAILABLE}

Or

Discovered group coordinator 172.25.1.2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:677)
(Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:509)
Group coordinator 172.25.1.2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:729)
Discovered group coordinator 172.25.40.219:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:677)
Group coordinator 172.25.1.2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:729)

Then, you need to configure listener settings like this in the kafka server.properties:

 listeners=PLAINTEXT://your.server.ip:9092

This is the solution tried on Apacke Kafka 2.5.0 and confluent platform 5.4.1.

This below line I have added in config/server.properties, that resolved my issue similar above issue. Hope this helps, its pretty much well documented in server.properties file, try to read and understand before you modify this. advertised.listeners=PLAINTEXT://<your_kafka_server_ip>:9092

For me, I didn't specify broker id for Kafka instance. It will get a new id from zookeeper sometimes when it restarts in Docker environment. If your broker id is greater than 1000, just specify the environment variable KAFKA_BROKER_ID.

Use this to see brokers, topics and partitions.

brew install kafkacat
kafkacat -b [kafka_ip]:[kafka_poot] -L

I was also getting the same error message

WARN Error while fetching metadata with correlation id 39 : {4-3-16-topic1=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Resolution Steps:

• Go to C:\Windows\System32\drivers\etc\hosts
• If the following line is not there, then add it to the end of the hosts file

127.0.0.1       localhost

• Go to C:\<Kafka_Config_Path>\server.properties, and, at the end of the file, add

    advertised.listeners = PLAINTEXT://localhost:9092
    listeners = PLAINTEXT://0.0.0.0:9092

• Restart the Kafka server

For all those struggling with the Kafka ssl setup and seeing this LEADER_NOT_AVAILABLE error. One of the reasons that might be broken is the keystore and truststore. In the keystore you need to have private key of the server + signed server certificate. In the client truststore, you need to have intermedidate CA certificate so that client can authenticate the kafka server. If you will use ssl for interbroker communication, you need this truststore also set in the server.properties of the brokers so they can authenticate each other.

That last piece I was mistakenly missing and caused me a lot of painful hours finding out what this LEADER_NOT_AVAILABLE error might mean. Hopefully this can help somebody.

Issue is resolved after adding the listener setting on server.properties file located at config directory. listeners=PLAINTEXT://localhost(or your server):9092 Restart kafka after this change. Version used 2.11

The advertised listeners as mentioned in the above answers could be one of the reason. The other possible reasons are:

1. The topic might not have been created. You can check this using bin/kafka-topics --list --zookeeper <zookeeper_ip>:<zookeeper_port>
2. Check your bootstrap servers that you have given to the producer to fetch the metadata. If the bootstrap server does not contain the latest metadata about the topic (for example, when it lost its zookeeper claim). You must be adding more than one bootstrap servers.

Also, ensure that you have the advertised listener set to IP:9092 instead of localhost:9092. The latter means that the broker is accessible only through the localhost.

When I encountered the error, I remember to have used PLAINTEXT://<ip>:<PORT> in the list of bootstrap servers (or broker list) and it worked, strangely.

bin/kafka-console-producer --topic sample --broker-list PLAINTEXT://<IP>:<PORT>

Try this listeners=PLAINTEXT://localhost:9092 It must be helpful

Many thanks

For me, it was happen due to a miss configuration
Docker port (9093)
Kafka command port "bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TopicName"
I checked my configuration to match port and now everything is ok

For me, the cause was using a specific Zookeeper that was not part of the Kafka package. That Zookeeper was already installed on the machine for other purposes. Apparently Kafka does not work with just any Zookeeper. Switching to the Zookeeper that came with Kafka solved it for me. To not conflict with the existing Zookeeper, I had to modify my confguration to have the Zookeeper listen on a different port:

[root@host /opt/kafka/config]# grep 2182 *
server.properties:zookeeper.connect=localhost:2182
zookeeper.properties:clientPort=2182

i know this was posted long time back, i would like to share how i solved it.
since i have my office laptop ( VPN and proxy was configured ).
i checked the environment variable NO_PROXY

> echo %NO_PROXY%

it returned with empty values
now i have set the NO_PROXY with localhost and 127.0.0.1

> set NO_PROXY=127.0.0.1,localhost  

if you want to append to existing values, then

> set NO_PROXY=%NO_PROXY%,127.0.0.1,localhost  

after this , i have restarted zookeeper and kafka
worked like a charm