Kafka connect plugins/connectors don't get installed properly using docker compose

Viewed 99

This is how my docker-compose yaml looks like:

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.1.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:6.1.0
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100

  schema-registry:
    image: confluentinc/cp-schema-registry:6.1.0
    container_name: schema-registry
    ports: 
      - 8081:8081
    depends_on:
      - zookeeper
      - broker
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: broker:29092

  kafka-connect:
    image: confluentinc/cp-kafka-connect-base:6.1.0
    container_name: kafka-connect
    depends_on:
      - zookeeper
      - broker
      - schema-registry
    ports:
      - 8083:8083
    environment:
      CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"
      CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO"
      CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
      CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/share/confluent-hub-components/'
    command: 
      - bash 
      - -c 
      - |
        echo "Installing connector plugins"
        confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.0.2
        confluent-hub install --no-prompt jcustenborder/kafka-connect-spooldir:2.0.60
        confluent-hub install --no-prompt streamthoughts/kafka-connect-file-pulse:1.5.0
        #
        # -----------
        # Launch the Kafka Connect worker
        /etc/confluent/docker/run &
        #
        # Don't exit
        sleep infinity
    volumes:
      - ${PWD}/data:/data

When I run docker-compose up -d it is all good but right after the containers are up, the one which holds schema registry exits. Then I start it once again and it works totally fine. The real problem is that when I execute the following command, I can not see any of my plugins installed:

curl -s localhost:8083/connector-plugins|jq '.[].class'|egrep 'SpoolDir'

I am following this tutorial more or less. I just skip the postgres container and also kafkacat and ksql.

EDIT This is what to logs say for the kafka-connect container:

Installing connector plugins
Running in a "--no-prompt" mode
Implicit acceptance of the license below:
https:/github.com/jcustenborder/kafka-connect-spooldir/LICENSE
Implicit confirmation of the question: You are about to install 'kafka-connect-spooldir' from Jeremy Custenborder, as published on Confluent Hub.
Downloading component Kafka Connect Spooldir 2.0.60, provided by Jeremy Custenborder from Confluent Hub and installing into /usr/share/confluent-hub-components
Adding installation directory to plugin path in the following files:
/etc/kafka/connect-distributed.properties
/etc/kafka/connect-standalone.properties
/etc/schema-registry/connect-avro-distributed.properties
/etc/schema-registry/connect-avro-standalone.properties

Completed

Running in a "--no-prompt" mode
Implicit acceptance of the license below:
The Apache Software License, Version 2.0
http://www.apache.org/licenses/LICENSE-2.0.txt
Implicit confirmation of the question: You are about to install 'kafka-connect-file-pulse' from StreamThoughts, as published on Confluent Hub.
Downloading component Kafka Connect File Pulse 1.5.0, provided by StreamThoughts from Confluent Hub and installing into /usr/share/confluent-hub-components
Adding installation directory to plugin path in the following files:
/etc/kafka/connect-distributed.properties
Adding installation directory to plugin path in the following files:
/etc/kafka/connect-distributed.properties
/etc/kafka/connect-standalone.properties
/etc/schema-registry/connect-avro-distributed.properties
/etc/schema-registry/connect-avro-standalone.properties

Completed

uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
1 Answers

The schema registry container may die because depends_on doesn't wait for the broker to start. Might just be a race condition. You should look at its logs rather than blindly restart it, though.

Your command is only looking for a connector class that includes SpoolDir.

If you want to see all installed connectors, don't pipe the curl command into anything and look at the Docker logs to see if any of the plugins failed to install

Related