How create kafka cluster and topics in docker-compose.yml

user_7116879 0
Viewed 1369

I am new to docker. I'd like containerize kafka and add a cluster, say MyCluster, to it and then add 4 topics, such as topic_a, topic_b, topic_c and topic_d to the cluster right after the kafka server starts. Can some please explain to me how to do that? I tried the following docker-compose.yml and I am able to launch containerized kafka server and kafka manager, but I can't create cluster/topics programmatically. Please help!

version: "2"

services:
  kafkaserver:
    image: "spotify/kafka:latest"
    container_name: kafka
    hostname: kafkaserver
    networks:
      - kafkanet
    ports:
      - 2181:2181
      - 9092:9092
    environment:
      ADVERTISED_HOST: kafkaserver
      ADVERTISED_PORT: 9092
    command:
      /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper kafkaserver:2181 --replication-factor 1 --partitions 1 --topic topic_a
      /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper kafkaserver:2181 --replication-factor 1 --partitions 1 --topic topic_b
      /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper kafkaserver:2181 --replication-factor 1 --partitions 1 --topic topic_c
      /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper kafkaserver:2181 --replication-factor 1 --partitions 1 --topic topic_d

  kafka_manager:
    image: "mzagar/kafka-manager-docker:1.3.3.4"
    container_name: kafkamanager
    networks:
      - kafkanet
    ports:
      - 9000:9000
    links:
      - kafkaserver
    environment:
      ZK_HOSTS: "kafkaserver:2181"

networks:
  kafkanet:
    driver: bridge
docker-compose apache-kafka docker
1 Answers

I am using this configuration at my docker-compose file. The whole file can be found here https://github.com/felipegutierrez/explore-flink/blob/master/operations-playground/docker-compose.yaml

  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    restart: always
  kafka:
    image: wurstmeister/kafka:2.12-2.2.1
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_CREATE_TOPICS: "input:2:1, output:2:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    ports:
      - 9094:9094
    restart: always
user_20969860
Related