I'm using the Debezium PostgreSQL connector to send my PostgreSQL data to Kafka. I set all configs correctly and it's working as expected in the local environment with docker-compose. Then we used Terraform to automate deployment to the AWS Fargate cluster. Terraform scripts also worked fine and launched all the required infrastructures. Then comes the problem;
- The connector doesn't start in Fargate and logs shows
GROUP_ID=1. ( This is set correctly in local with docker-composeGROUP_ID=connect-group-dev) - I provide the
GROUP_IDasconnect-group-devin environment variables but that is not reflected in to the Fargate cluster container, however in the AWS UI, I can see thatGROUP_IDis set toconnect-group-dev. - All other environment variables are reflected in to the container.
I suspect the problem is that GROUP_ID is not getting by the container when it's starting the Kafka Connector, but in a later step, it is set to the container. ( because I can see the correct value in AWS UI in the Task Definition )
Is the default value is 1 for GROUP_ID? (since I don't set any variable to 1 )
This is a weird situation and double-check all the files, but still cannot find a reason for this. Any help would be great.