Facing error response from daemon-Windows

Question

I am trying to run apache Kafka on windows using docker and my docker-compose.yml code is as follows:

version: "3"
services:
  spark:
    image: jupyter/pyspark-notebook
    ports:
      - "9092:9092"
      - "4010-4109:4010-4109"
    volumes:
      - ./notebooks:/home/jovyan/work/notebooks/
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    container_name: zookeeper
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    container_name: kakfa
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

When I execute the command

docker-compose -f docker-compose.yml up

I get an error: Error response from daemon: driver failed programming external connectivity on endpoint kafka-spark-1 (452eae1760b7860e3924c0e630943f825a809272760c8aa8bbb2f58ab2865377): Bind for 0.0.0.0:9092 failed: port is already allocated

I have tried net stop winnat and net start winnat, unfortunately this solution didn't work. Would appreciate any kind of help!

OneCricketeer · Answer

Spark isn't running Kafka

Remove the ports here

image: jupyter/pyspark-notebook
ports:
  - "9092:9092"

Also, change variable for Kafka to use the proper hostname, otherwise Spark will not work with it...

KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092

Then you can also remove ports for Kafka container since you wouldn't have access from the host. Unless you add external listeners.

You may also be interested in an example notebook I use to test PySpark with Kafka.

Facing error response from daemon-Windows

1 Answers