Running a distributed kafka connect cluster on AWS ECS

user_7499004 0
Viewed 476

I want to run a kafka connect multi node cluster in ECS (ec2 type).

In order for kafka-connect nodes in the same cluster to communicate I need to set up the REST_ADVERTISED_HOST_NAME property to be the ip / hostname of the ec2 instance that the ecs task is running on.

This property should be different for each kafka-connect node (running as ECS task)

when I increase the number of tasks in the ECS service, how can I setup a different REST_ADVERTISED_HOST_NAME for every task? under the hood, the tasks could be running on the same ec2 instance

Currently this property is hardcoded in the connect-distributed.properties or passed as an environment variable in the task definition, and so its the same for every new task (or connect node) that is created

apache-kafka-connect apache-kafka docker amazon-web-services
2 Answers

you need a load balancer that is accessible from all tasks/nodes.

user_71293480

This question is pretty old, but we ran into this issue and solved it by introducing a custom entrypoint that sets the CONNECT_REST_ADVERTISED_HOST_NAME during boot-up.

Our Dockerfile (simplified):

FROM confluentinc/cp-server-connect-base:7.2.1

COPY entrypoint.sh .

And entrypoint.sh:

#!/bin/bash

export CONNECT_REST_ADVERTISED_HOST_NAME=$HOSTNAME

/etc/confluent/docker/run
user_6312380
Related