kafka docker stuck on configuration and not working on mac

user_7804550 0
Viewed 1981

I am trying to use kafka on docker. But when running the docker compose, kafka just stuck on configuring mode and not indicating that already running. I do listing for kafka topics but it's not working or not giving the response. Here's my docker-compose.yml:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181
    environment:
      ZOO_MY_ID: 1
    networks:
      - kafka_net
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - 9092:9092
    expose:
      - 9092
    depends_on:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 1
    restart: always
    networks:
      - kafka_net
networks:
  kafka_net:
    driver: "bridge"

Here's the zookeeper response on docker-compose log:

2021-04-19 01:07:56,385 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/opt/zookeeper-3.4.13
2021-04-19 01:07:56,393 [myid:] - INFO [main:ZooKeeperServer@836] - tickTime set to 2000
2021-04-19 01:07:56,394 [myid:] - INFO [main:ZooKeeperServer@845] - minSessionTimeout set to -1
2021-04-19 01:07:56,394 [myid:] - INFO [main:ZooKeeperServer@854] - maxSessionTimeout set to -1
2021-04-19 01:07:56,418 [myid:] - INFO [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2021-04-19 01:07:56,430 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181

Here's the only kafka response on docker-compose log:

kafka        | [Configuring] 'log.dirs' in '/opt/kafka/config/server.properties'
kafka        | [Configuring] 'zookeeper.connect' in '/opt/kafka/config/server.properties'
kafka        | [Configuring] 'listeners' in '/opt/kafka/config/server.properties'
kafka        | Excluding KAFKA_VERSION from broker config
kafka        | [Configuring] 'broker.id' in '/opt/kafka/config/server.properties'
kafka        | [Configuring] 'listener.security.protocol.map' in '/opt/kafka/config/server.properties'
kafka        | [Configuring] 'advertised.listeners' in '/opt/kafka/config/server.properties'
kafka        | [Configuring] 'port' in '/opt/kafka/config/server.properties'
kafka        | [Configuring] 'advertised.host.name' in '/opt/kafka/config/server.properties'
kafka        | Excluding KAFKA_HOME from broker config
kafka        | [Configuring] 'advertised.port' in '/opt/kafka/config/server.properties'
kafka        | [Configuring] 'inter.broker.listener.name' in '/opt/kafka/config/server.properties'

When running this command, it's not giving a response:

docker exec -it kafka /bin/sh
/ # cd /opt/kafka_2.13-2.7.0
/opt/kafka_2.13-2.7.0 # bin/kafka-topics.sh --list --zookeeper zookeeper:2181

what should i do on this? i still trying to make it run. I already try with kafka native (without docker) and it run perfectly the i can create a topic

kafka-consumer-api apache-kafka docker
2 Answers

Try to use another image of kafka, to example - bitnami. I had the same problem as you on Apple M1 Big Shur, and i used bitnami and it's working!! this is my docker-compose.yml:

version: "2"

services:
    zookeeper:
        image: docker.io/bitnami/zookeeper:3
        ports:
            - "2181:2181"
        volumes:
            - "zookeeper_data:/bitnami"
        environment:
            - ALLOW_ANONYMOUS_LOGIN=yes
    kafka:
        image: docker.io/bitnami/kafka:2
        ports:
            - "9092:9092"
        volumes:
            - "kafka_data:/bitnami"
        environment:
            - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
            - ALLOW_PLAINTEXT_LISTENER=yes
            - KAFKA_LISTENERS=PLAINTEXT://:9092
            - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
        depends_on:
            - zookeeper

volumes:
    zookeeper_data:
        driver: local
    kafka_data:
        driver: local
user_160198270

Seems like you missed an important part from your docker-compose log. I ran your docker-compose file on my local and would like to highlight this part from the logs.

kafka        | [2021-04-19 09:20:19,209] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
kafka        | [2021-04-19 09:20:19,253] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
kafka        | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9092
kafka        |  at kafka.utils.CoreUtils$.validate$1(CoreUtils.scala:265)
kafka        |  at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:276)
kafka        |  at kafka.server.KafkaConfig.$anonfun$listeners$1(KafkaConfig.scala:1680)
kafka        |  at kafka.server.KafkaConfig.listeners(KafkaConfig.scala:1679)
kafka        |  at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1779)
kafka        |  at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1756)
kafka        |  at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1312)
kafka        |  at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
kafka        |  at kafka.Kafka$.main(Kafka.scala:68)
kafka        |  at kafka.Kafka.main(Kafka.scala)
kafka exited with code 1

So you were using the same port for accessing Kafka from inside and outside the docker network. I changed the below setting (the INSIDE port) and it worked for me. I was able to create and list topics. I was also able to produce and consume simple messages from the kafka container.

 KAFKA_LISTENERS: INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
 KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:29092,OUTSIDE://localhost:9092

This is some of the testing I did from the kafka container

docker exec -it kafka bash    

bash-4.4# cd /opt/kafka_2.13-2.7.0/
bash-4.4# bin/kafka-topics.sh --create --topic test-topic --partitions 16 --replication-factor 1 --zookeeper zookeeper:2181
Created topic test-topic.

bash-4.4# bin/kafka-topics.sh --list --zookeeper zookeeper:2181
test-topic

This is an excellent post that I would like to refer you. Please let me know if this works for you or not.

user_43624170
Related