Kafka cannot create topic using docker

user_5684848 0
Viewed 2754

I am attempting to use docker to test our Kafka listeners

Both of the servers seem to start correctly as I am able to run the command

docker exec -it bsbecpromoeventservice_broker_run_ping zookeeper
PING zookeeper (172.20.0.2): 56 data bytes
64 bytes from 172.20.0.2: icmp_seq=0 ttl=64 time=0.089 ms
64 bytes from 172.20.0.2: icmp_seq=1 ttl=64 time=0.193 ms
64 bytes from 172.20.0.2: icmp_seq=2 ttl=64 time=0.201 ms
^C--- zookeeper ping statistics ---

However when I try to run the following command

docker exec -it bsbecpromoeventservice_broker_run_4 kafka-topics --create --zookeeper zookeeper:2 -replication-factor 1 --partitions 1 --topic test

I get 

[2017-09-13 15:22:43,967] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

Here is my docker-compose file

version: '3'
services:
  zookeeper:
    hostname: zookeeper
    image: confluentinc/cp-zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-enterprise-kafka
    hostname: broker
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:9092'
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:9092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

  test:
    image: java:8
    volumes:
        - .:/src
        - ${GRADLE_USER_HOME}:/gradle_mount
    environment:
      - GRADLE_USER_HOME=/gradle_mount
      - SPRING_KAFKA_BOOTSTRAP-SERVERS=broker:9092
    depends_on:
        - broker
        - zookeeper
    working_dir: /src
    privileged: true
    command:  "./gradlew --no-daemon clean test --info"

How can I make it so that the Kafka server is able to create topics?

docker-compose apache-kafka docker
2 Answers

Container run command to create topic

Use the container command docker run --net=host --rm. In the following example, the zookeeper is running on port 22181, please use the respective topic name, port.

Create

docker run   --net=host   --rm   confluentinc/cp-kafka:4.0.0   kafka-topics --create --topic customer --partitions 1 --replication-factor 1 --if-not-exists --zookeeper  localhost:22181

Describe

docker run   --net=host   --rm   confluentinc/cp-kafka:4.0.0   kafka-topics --zookeeper localhost:22181 --topic customer --describe

List

docker run   --net=host   --rm   confluentinc/cp-kafka:4.0.0   kafka-topics --list --zookeeper  localhost:22181

Delete

docker run   --net=host   --rm   confluentinc/cp-kafka:4.0.0   kafka-topics  --delete --topic customer --zookeeper localhost:22181
user_43488240

If you use version older than 2.2 you should use --zookeeper option and pass connection string to zookeeper

Command would be something like that:

docker exec -it <zookeeper_container_id> kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testLogs

docker exec -it <zookeeper_container_id> kafka-topics --list --zookeeper localhost:2181
user_35739380
Related