How to install connectors to the docker image of apache kafka connect

user_10254670 0
Viewed 2538

I am using https://hub.docker.com/r/1ambda/kafka-connect/ to install docker kafka connect on ubuntu .I am able to run it but ,I am unable to install any more connectors in it.

What I have tried

1-I tried copying the connector files from my machine to the docker containers connector folder and restarting but the api ,http://localhost:8080/connectors give empty array.

2-Also added plugin path in connect-distributed.properties . How to do this any idea ??

apache-kafka-connect apache-kafka docker
3 Answers

Solution without using Confluent Hub

Step 1: Build your own Kafka-Connect image

Your directory should look like this:

my-directory /
  Dockerfile
  plugins /
    my-connector /
       <connector-jars>

Dockerfile:

FROM confluentinc/cp-kafka-connect-base:latest
USER root:root
COPY ./plugins/ /opt/kafka/plugins/
ENV CONNECT_PLUGIN_PATH="/opt/kafka/plugins"
USER 1001

Run the following

cd /my-directory
sudo docker build . -t my-connector-image-name

Step 2: Use the created image for your kafka-connect container

version: '3'
services:
        zookeeper:
                container_name: 'zookeeper'
                image: 'bitnami/zookeeper:latest'
                ports:
                        - '2181:2181'
                environment:
                        - ALLOW_ANONYMOUS_LOGIN=yes
        kafka:
                image: 'bitnami/kafka:latest'
                container_name: 'kafka'
                ports:
                        - '9092:9092'
                environment:
                        - KAFKA_BROKER_ID=1
                        - KAFKA_LISTENERS=PLAINTEXT://:9092
                        - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.178.70:9092
                        - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
                        - ALLOW_PLAINTEXT_LISTENER=yes
                depends_on:
                        - zookeeper
        kafka-connect:
                image: 'my-connector-image-name:latest'
                container_name: 'kafka-connect'
                ports:
                        - '8083:8083'
                environment:
                        - CONNECT_BOOTSTRAP_SERVERS=kafka:9092
                        - CONNECT_REST_PORT=8083
                        - CONNECT_GROUP_ID=quickstart
                        - CONNECT_CONFIG_STORAGE_TOPIC=quickstart-config
                        - CONNECT_OFFSET_STORAGE_TOPIC=quickstart-offsets
                        - CONNECT_STATUS_STORAGE_TOPIC=quickstart-status
                        - CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_REST_ADVERTISED_HOST_NAME=localhost
                depends_on:
                        - kafka

Step 3 Get available Connector Plugins

curl localhost:8083/connector-plugins | json_pp

Side note on my Kafka configuration: I'm hosting Docker inside a Linux virtual machine that I've assigned IP 192.168.178.70.

user_97764040
user_23086830

1-I tried copying the connector files from my machine to the docker containers connector folder and restarting but the api ,http://localhost:8080/connectors give empty array.

localhost:8083/connectors , will give you the list of active connectors. If you need to check the list of available plugins you should hit localhost:8083/connector-plugins

curl localhost:8083/connector-plugins
[{"class":"io.confluent.connect.activemq.ActiveMQSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector","type":"sink","version":"5.5.1"},{"class":"io.confluent.connect.ibm.mq.IbmMQSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.jdbc.JdbcSinkConnector","type":"sink","version":"5.5.1"},
{"class":"io.confluent.connect.jdbc.JdbcSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.jms.JmsSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.s3.S3SinkConnector","type":"sink","version":"5.5.1"},{"class":"io.confluent.connect.storage.tools.SchemaSourceConnector","type":"source","version":"5.5.1-ccs"},{"class":"io.confluent.kafka.connect.datagen.DatagenConnector","type":"source","version":"null"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"5.5.1-ccs"}, 
{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"source","version":"5.5.1-ccs"},{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"1"}]

Also, a new connector needs to be added to the mentioned plugin-path of all your worker nodes and a restart is required for all

To further troubleshoot you can try increasing the loglevel to DEBUG and check the startup logs if the connector path is being traversed You will see a log with the following format :

DEBUG Loading plugin urls: ......

Hope this helps resolve your issue :)

user_126127780
Related