Confluent Kafka connect distributed mode jdbc connector

user_6465598 0
Viewed 1367

We have successfully used MySQL - kafka data ingestion using jdbc standalone connector but now facing issue in using the same in distributed mode (as kafka connect service ).

Command used for standalone connector which works fine - 

/usr/bin/connect-standalone /etc/kafka/connect-standalone.properties /etc/kafka-connect-jdbc/source-quickstart-mysql.properties

Now we have stopped this one and started the kafka connect service in distributed mode like this -

systemctl status confluent-kafka-connect
● confluent-kafka-connect.service - Apache Kafka Connect - distributed
   Loaded: loaded (/usr/lib/systemd/system/confluent-kafka-connect.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-11-14 22:52:49 CET; 41min ago
     Docs: http://docs.confluent.io/
 Main PID: 130178 (java)
   CGroup: /system.slice/confluent-kafka-connect.service
           └─130178 java -Xms256M -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.a...

2 nodes are currently running the connect service with same connect-distributed.properties file .

bootstrap.servers=node1IP:9092,node2IP:9092
group.id=connect-cluster
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.topic=connect-offsets
offset.storage.replication.factor=1
config.storage.topic=connect-configs
config.storage.replication.factor=1
status.storage.topic=connect-status
status.storage.replication.factor=1
offset.flush.interval.ms=10000
plugin.path=/usr/share/java

The connect service is UP and running but it doesn't load the connectors defined under /etc/kafka/connect-standalone.properties.

What should be done to the service so that whenever you hit the command systemctl start confluent-kafka-connect , it runs the service and starts the defined connectors under /etc/kafka-connect-*/ just like when you run a standalone connector manually providing paths to properties files.

confluent-platform apache-kafka-connect apache-kafka
3 Answers

it runs the service and starts the defined connectors under /etc/kafka-connect-*/

That's not how distributed mode works... It doesn't know which property files you want to load, and it doesn't scan those folders1

With standalone-mode the N+1 property files that you give are loaded immediately, yes, but for connect-distributed, you must use HTTP POST calls to the Connect REST API.

Confluent Control Center or Landoop's Connect UI can provide a nice management web portal for these operations.

By the way, if you have more than one broker, I'll suggest increasing the replica factors on those connect topics in the connect-distributed.properties file.

1. It might be a nice feature if it did, but then you have to ensure connectors are never deleted/stopped in distributed mode, and you just end up in an inconsistent state with what's running and the files that are on the filesystem.

user_23086830

I can describe what I did for starting the jdbc connector in distributed mode:

I am using on my local machine, confluent CLI utility for booting up faster the services.

./confluent start

Afterwords I stopped kafka-connect

./confluent stop connect

and then I proceed to manually start the customized connect-distributed on two different ports (18083 and 28083)

➜  bin ./connect-distributed ../etc/kafka/connect-distributed-worker1.properties

➜  bin ./connect-distributed ../etc/kafka/connect-distributed-worker2.properties

NOTE: Set plugin.path setting to the full (and not relative) path (e.g.: plugin.path=/full/path/to/confluent-5.0.0/share/java)

Then i can easily add a new connector 

curl -s -X POST -H "Content-Type: application/json" --data @/full/path/to/confluent-5.0.0/etc/kafka-connect-jdbc/source-quickstart-sqlite.json http://localhost:18083/connectors

This should do the trick.

As already pointed out by cricket_007 consider a replication factor of at least 3 for the kafka brokers in case you're dealing with stuff that you don't want to lose in case of outage of one of the brokers.

user_4977940
user_119331690
Related