Kafka connect consumer Group ID doesn't set in Fargate cluster

Viewed 12

I'm using the Debezium PostgreSQL connector to send my PostgreSQL data to Kafka. I set all configs correctly and it's working as expected in the local environment with docker-compose. Then we used Terraform to automate deployment to the AWS Fargate cluster. Terraform scripts also worked fine and launched all the required infrastructures. Then comes the problem;

  • The connector doesn't start in Fargate and logs shows GROUP_ID=1. ( This is set correctly in local with docker-compose GROUP_ID=connect-group-dev )
  • I provide the GROUP_ID as connect-group-dev in environment variables but that is not reflected in to the Fargate cluster container, however in the AWS UI, I can see that GROUP_ID is set to connect-group-dev.
  • All other environment variables are reflected in to the container.

I suspect the problem is that GROUP_ID is not getting by the container when it's starting the Kafka Connector, but in a later step, it is set to the container. ( because I can see the correct value in AWS UI in the Task Definition )

Is the default value is 1 for GROUP_ID? (since I don't set any variable to 1 )

This is a weird situation and double-check all the files, but still cannot find a reason for this. Any help would be great.

0 Answers
Related