Setting up Kafka Connect, cannot rename group ID

Question

I am using this Github repo and folder path I found: https://github.com/entechlog/kafka-examples/tree/master/kafka-connect-standalone

The issue I am having is that, as a matter of access control, I must specify my group ID by adding a prefix to it, let's call it abc-. When I build this Docker image, I check my logs and I can see that the group ID ends up being connect-bq-sink-connector, which I am assuming is a concatenation of the word connect- along with the variable CONNECTOR_NAME seen in the docker-compose file. When I change the connector name variable, my group ID also changes (but the connect- prefix always remains). You will also see a variable called CONNECT_GROUP_ID in the docker-compose file. This variable appears to have absolutely no effect on the Kafka connect instance. The Docker logs give this (in this order):

WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:380)

and then later...

...

group.id = connect-bq-sink-connector

The final error, which is mostly unimportant as I know it is due to lack of permissions, is simply:

[2021-12-03 22:37:32,775] INFO [bq-sink-connector|task-0] [Consumer clientId=connector-consumer-bq-sink-connector-0, groupId=connect-bq-sink-connector] FindCoordinator request hit fatal exception (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:260)

org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access Group: connect-bq-sink-connector

and so two things: I cannot figure out why the word connect- is being prefixed every time to the resulting group ID, and also why the variable CONNECT_GROUP_ID seems to not function at all.

I thought maybe the image is forcibly hard coding connect- in front. I tried running this by changing line 50 in the docker compose file from image: entechlog/cp-kafka-connect${CONFLUENT_VERSION_TAG} to image: confluentinc/cp-kafka:5.0.0 and it changed nothing.

What is causing this prefix to be added, and how can I override this? I can keep the word connect there if necessary, but I simply need to add my prefix abc- in front of it.

user_2308683 · Answer

The Sink Connector consumer group id prefix cannot be changed

https://issues.apache.org/jira/browse/KAFKA-4400

CONNECT_GROUP_ID is only applicable for Kafka Connect distributed mode clustering, not consumers, which the Confluent images use. I don't see any benefit of using the linked image that runs standalone since that would fail to be fault tolerant or scalable (the Connect configs and status are stored in an ephemeral filesystem rather than back in Kafka)

user_17577431 · Answer

If you want to change connect group id, add environment variable name CONNECTOR_ properties section under the service kafka-connect and set a value you want.

The github example starting steps as follows.

In file docker/Dockerfile, a startup command is /etc/confluent/docker/run and you cant find the file in docker/include/etc/confluent/docker.
Start a container with simple step configure and launch in the docker/include/etc/confluent/docker/run file.
In file docker/include/etc/confluent/docker/configure, Check mandatory environment variables such as CONNECT_BOOTSTRAP_SERVERS, CONNECT_KEY_CONVERTER, CONNECT_VALUE_CONVERTER... are set, and call templating function with kafka-connect-standalone.properties.template and kafka-connect.properties.template.

So if there is a configuration that you want to add to the kafka-connect-standalone.properties file, you must specify an environment variable starting with CONNECTOR_.

You can find all configuration for kafka connect in the following link.

https://kafka.apache.org/documentation/#connectconfigs

user_14032355 · Answer

Currently, there is no way to change the group id for the kafka sink connector.

consumer group id is hardcoded and prefixed with connect-.

package org.apache.kafka.connect.util;

public final class SinkUtils {

    private SinkUtils() {}

    public static String consumerGroupId(String connector) {
        return "connect-" + connector;
    }

}

Ref. https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/util/SinkUtils.java

Setting up Kafka Connect, cannot rename group ID

3 Answers