toxiproxy for kafka in a docker-compose file

Question

I'm trying to integrate toxy-proxy with a kafka cluster, but seems to not work.

The docker-compose file looks something like this:

kafka:
image: confluentinc/cp-kafka:latest
depends_on:
  - zookeeper
environment:
  KAFKA_BROKER_ID: 1
  KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182
  KAFKA_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
  KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

toxiproxy:
image: 'ghcr.io/shopify/toxiproxy:2.4.0'
ports:
  - 8474:8474
  - 29092:29092

Also my application is set up like this:

my_application:
image: image
environment:
  KAFKA_BROKERS_URL: toxiproxy:8474

The problem is that my application does not connect to kafka. Not sure how should be done. I used this https://github.com/Shopify/sarama/blob/main/docker-compose.yml as an example. Also I'd like to know how to call kafka from my localhost, so not from inside a docker container.

Basically this works now

kafka:
image: confluentinc/cp-kafka:latest
depends_on:
  - zookeeper
ports:
  - 29092:29092
environment:
  KAFKA_BROKER_ID: 1
  KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
  KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1


  toxiproxy:
container_name: toxiproxy
image: 'ghcr.io/shopify/toxiproxy:latest'
command: ['-config', '/config/toxiproxy.json','-host', '0.0.0.0']
volumes:
  - ./config/toxiproxy.json:/config/toxiproxy.json
ports:
  - target: 8474
    published: 8474
    protocol: tcp
    mode: host
depends_on:
  - kafka

Where the config file looks like this:

[
{
    "name": "kafka_proxy",
    "listen": "[::]:45390",
    "upstream": "kafka:9092",
    "enabled": true
}

]

However because after the application makes a first connection to Kafka it will ask for cluster configuration, and will be served hostnames according to ADVERTISED_LISTENERS, which will be set to the actual kafka container and not to my proxy.

This I still don't know how to do, to set the advertised_listeners to use the proxy as well. I've tried setting it like this, but kafka stops working.

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://toxiproxy:45390

toxiproxy for kafka in a docker-compose file

0 Answers