not able to connect with the kafka in docker using docker-file

user_8085276 0
Viewed 1405
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.5.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - ./.docker-volumes/zookeeper/_data:/var/lib/zookeeper/data
      - ./.docker-volumes/zookeeper/_log:/var/lib/zookeeper/log

  kafka:
    image: confluentinc/cp-kafka:5.5.1
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./.docker-volumes/kafka/_data:/var/lib/kafka/data

Above is the docker-compose file to spin up the zookeeper and kafka instances in docker.

When I run the service which is trying to make a connection with the kafka. I'm getting below error. I tracked it down on the internet but did not get much on this.

Error on the Service-layer

.12:43:18.730 [kafka-producer-network-thread | producer-2] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Connection to node -1 (localhost/127.0.0.1:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. 
.12:43:19.295 [kafka-producer-network-thread | producer-1] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. 
.12:43:19.481 [kafka-admin-client-thread | vendor_srs_consumer_srs_restaurant_finance_pan_validate_client_id_0] WARN  o.apache.kafka.clients.NetworkClient - [AdminClient clientId=vendor_srs_consumer_srs_restaurant_finance_pan_validate_client_id_0] Connection to node -1 (localhost/127.0.0.1:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

Error in Docker when kafka-producer is trying to connect with the kafka brokers.

[2021-07-12 07:13:19,492] WARN [SocketServer brokerId=1] Unexpected error from /172.20.0.1; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369296129 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:447)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:397)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at kafka.network.Processor.poll(SocketServer.scala:893)
at kafka.network.Processor.run(SocketServer.scala:792)
at java.lang.Thread.run(Thread.java:748)

Not sure why producer is not able to connect with the broker. Thanks in advance.

kafka-producer-api docker-compose apache-kafka docker java
2 Answers

PLAINTEXT_HOST://0.0.0.0:9092 is not a valid advertised listener; this needs to be a resolvable address.

In your error, Connection to node -1 (localhost/127.0.0.1:9092) terminated, unclear where this code is running, but if on the same host outside of Docker, then you should advertise localhost, not 0.0.0.0

Your other error, InvalidReceiveException: Invalid receive (size = 369296129 larger than 104857600) indicates that you have connected to some other server and got a response that the client did not expect.

user_23086830

Initially they are on default bridge network, and in order to communicate between the containers you have to use container IPs.

You can try deploying both of the containers on same network.

Add a network definition in your docker-compose

networks:
  default:
    external:
      name: zookeeper-kafka
  zookeeper-kafka:
    driver: bridge

In services definition you can add network name and ports.

Example, for zookeeper service

zookeeper:
    networks:
    - zookeeper-kafka
    image: confluentinc/cp-zookeeper:5.5.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - ./.docker-volumes/zookeeper/_data:/var/lib/zookeeper/data
      - ./.docker-volumes/zookeeper/_log:/var/lib/zookeeper/log

And then you can access zookeeper service via zookeeper-kafka:2181 (i.e. network-name:port), while connecting through kafka.

Example for update kafka service definition

kafka:
    networks:
    - zookeeper-kafka
    image: confluentinc/cp-kafka:5.5.1
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-kafka:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://zookeeper-kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./.docker-volumes/kafka/_data:/var/lib/kafka/data

Hopefully this helps you

user_133383440
Related