I'm implementing a change data capture scenario using Azure Event Hub (kafka surface enabled), to capture data from postgresql. The database is Postgresql and uses the debizum connector (https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.2.0.Final/debezium-connector-postgres-1.2.0.Final-plugin.tar.gz). I also used azure postgres database (single server- logical replication) as well. The kafka connect runs on the docker locally, and can create topics in the azure event hubs (the docker compose file attached). Also once I send the rest request to the kafka connect, it shows the connector is up and running. However, when I insert data into the postgres table, it cannot create the topic in the eventhub, and I could not figure out the reason?
Image of topics created for the kafka connect in eventhub
The status of connector:
localhost:8083/connectors/postgres-connector/status
{
"name": "postgres-connector",
"connector": {
"state": "RUNNING",
"worker_id": "connect:8083"
},
"tasks": [
{
"id": 0,
"state": "RUNNING",
"worker_id": "connect:8083"
}
],
"type": "source"
}
This also looks weird to me that as soon as creating the source connector the kafka connect shows the following warning regarding connection to eventhub:
docker-file-connect-1 | [2022-09-22 08:24:11,152] INFO [Producer clientId=connector-producer-postgres-connector-0] Cancelled in-flight API_VERSIONS request with correlation id 32338 due to node -1 being disconnected (elapsed time since creation: 8ms, elapsed time since send: 8ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient) docker-file-connect-1 | [2022-09-22 08:24:11,152] WARN [Producer clientId=connector-producer-postgres-connector-0] Bootstrap broker eventhubstandard.servicebus.windows.net:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
the post request for creating the connector:
localhost:8083/connectors
{
"name": "postgres-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "postgres",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname": "postgres",
"database.server.name": "todos-server",
"plugin.name": "wal2json",
"table.whitelist": "public.todos"
}
}
the docker file used for connect and postgres
version: '2'
services:
connect:
image: mtpatter/debezium-connect # built from debezium/connect:0.10
hostname: connect
image: confluentinc/cp-server-connect-base:latest
ports:
- '8083:8083'
environment:
CONNECT_BOOTSTRAP_SERVERS: 'eventhubstandard.servicebus.windows.net:9093'
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: connect-cluster
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_CONFIG_STORAGE_TOPIC: connect_config_topic
CONNECT_OFFSET_STORAGE_TOPIC: connect_offset_topic
CONNECT_STATUS_STORAGE_TOPIC: connect_status_topic
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
# Connect Worker
CONNECT_SECURITY_PROTOCOL: SASL_SSL
CONNECT_SASL_MECHANISM: PLAIN
CONNECT_SASL_JAAS_CONFIG: "XXXXX"
# Connect Producer
CONNECT_SECURITY_PROTOCOL: SASL_SSL
CONNECT_PRODUCER_SASL_MECHANISM: PLAIN
CONNECT_PRODUCER_SASL_JAAS_CONFIG: "XXXXXX"
CONNECT_SECURITY_PROTOCOL: SASL_SSL
CONNECT_SASL_MECHANISM: PLAIN
CONNECT_CONSUMER_SASL_JAAS_CONFIG: "XXXXXXX"
CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_INTERNAL_KEY_CONVERTER_SCHEMAS_ENABLE: "false"
CONNECT_INTERNAL_VALUE_CONVERTER_SCHEMAS_ENABLE: "false"
command:
- bash
- -c
- |
echo "Installing connector plugins"
confluent-hub install --no-prompt debezium/debezium-connector-sqlserver:latest
confluent-hub install --no-prompt debezium/debezium-connector-postgresql:latest
#
echo "launching kafka connector worker"
/etc/confluent/docker/run &
#
sleep infinity
volumes:
- './kafka_connect:/connect'