Azure Event hub cannot create topics for change data capture while integrating with kafka connect, debeizum connector and postgrsql

Viewed 10

I'm implementing a change data capture scenario using Azure Event Hub (kafka surface enabled), to capture data from postgresql. The database is Postgresql and uses the debizum connector (https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.2.0.Final/debezium-connector-postgres-1.2.0.Final-plugin.tar.gz). I also used azure postgres database (single server- logical replication) as well. The kafka connect runs on the docker locally, and can create topics in the azure event hubs (the docker compose file attached). Also once I send the rest request to the kafka connect, it shows the connector is up and running. However, when I insert data into the postgres table, it cannot create the topic in the eventhub, and I could not figure out the reason?

Image of topics created for the kafka connect in eventhub

The status of connector:

localhost:8083/connectors/postgres-connector/status

{
    "name": "postgres-connector",
    "connector": {
        "state": "RUNNING",
        "worker_id": "connect:8083"
    },
    "tasks": [
        {
            "id": 0,
            "state": "RUNNING",
            "worker_id": "connect:8083"
        }
    ],
    "type": "source"
}

This also looks weird to me that as soon as creating the source connector the kafka connect shows the following warning regarding connection to eventhub:

docker-file-connect-1 | [2022-09-22 08:24:11,152] INFO [Producer clientId=connector-producer-postgres-connector-0] Cancelled in-flight API_VERSIONS request with correlation id 32338 due to node -1 being disconnected (elapsed time since creation: 8ms, elapsed time since send: 8ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient) docker-file-connect-1 | [2022-09-22 08:24:11,152] WARN [Producer clientId=connector-producer-postgres-connector-0] Bootstrap broker eventhubstandard.servicebus.windows.net:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

the post request for creating the connector:

localhost:8083/connectors

{
    "name": "postgres-connector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "database.hostname": "postgres",
        "database.port": "5432",
        "database.user": "postgres",
        "database.password": "postgres",
        "database.dbname": "postgres",
        "database.server.name": "todos-server",
        "plugin.name": "wal2json",
        "table.whitelist": "public.todos"
    }
}

the docker file used for connect and postgres

version: '2'
services:
  connect:
    image: mtpatter/debezium-connect  # built from debezium/connect:0.10 
    hostname: connect
    image: confluentinc/cp-server-connect-base:latest
    ports:
    - '8083:8083'
    environment:
      CONNECT_BOOTSTRAP_SERVERS: 'eventhubstandard.servicebus.windows.net:9093'
      CONNECT_REST_ADVERTISED_HOST_NAME: connect
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: connect-cluster

      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_CONFIG_STORAGE_TOPIC: connect_config_topic
      CONNECT_OFFSET_STORAGE_TOPIC: connect_offset_topic
      CONNECT_STATUS_STORAGE_TOPIC: connect_status_topic
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      # Connect Worker
      CONNECT_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_SASL_MECHANISM: PLAIN
      CONNECT_SASL_JAAS_CONFIG: "XXXXX"
      # Connect Producer
      CONNECT_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_PRODUCER_SASL_MECHANISM: PLAIN
      CONNECT_PRODUCER_SASL_JAAS_CONFIG: "XXXXXX"
      CONNECT_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_SASL_MECHANISM: PLAIN
      CONNECT_CONSUMER_SASL_JAAS_CONFIG: "XXXXXXX"
      CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_INTERNAL_KEY_CONVERTER_SCHEMAS_ENABLE: "false"
      CONNECT_INTERNAL_VALUE_CONVERTER_SCHEMAS_ENABLE: "false"
    command:
      - bash
      - -c
      - |
        echo "Installing connector plugins"
        confluent-hub install --no-prompt debezium/debezium-connector-sqlserver:latest
        confluent-hub install --no-prompt debezium/debezium-connector-postgresql:latest
        #
        echo "launching kafka connector worker"
        /etc/confluent/docker/run &
        #
        sleep infinity
    volumes:
      - './kafka_connect:/connect'
0 Answers
Related