What is a simple, effective way to debug custom Kafka connectors?

user_8418099 0
Viewed 20811

I'm working a couple of Kafka connectors and I don't see any errors in their creation/deployment in the console output, however I am not getting the result that I'm looking for (no results whatsoever for that matter, desired or otherwise). I made these connectors based on Kafka's example FileStream connectors, so my debug technique was based off the use of the SLF4J Logger that is used in the example. I've searched for the log messages that I thought would be produced in the console output, but to no avail. Am I looking in the wrong place for these messages? Or perhaps is there a better way of going about debugging these connectors?

Example uses of the SLF4J Logger that I referenced for my implementation:

Kafka FileStreamSinkTask

Kafka FileStreamSourceTask

apache-kafka-connect apache-kafka slf4j debugging java
3 Answers

i love the accepted answer. one thing - the environment variables didn't work for me... i'm using confluent community edition 5.3.1...

here's what i did that worked...

i installed the confluent cli from here: https://docs.confluent.io/current/cli/installing.html#tarball-installation

i ran confluent using the command confluent local start

i got the connect app details using the command ps -ef | grep connect

i copied the resulting command to an editor and added the arg (right after java):

-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

then i stopped connect using the command confluent local stop connect

then i ran the connect command with the arg

brief intermission ---

vs code development is led by erich gamma - of gang of four fame, who also wrote eclipse. vs code is becoming a first class java ide see https://en.wikipedia.org/wiki/Erich_Gamma

intermission over ---

next i launched vs code and opened the debezium oracle connector folder (cloned from here) https://github.com/debezium/debezium-incubator

then i chose Debug - Open Configurations

enter image description here

and entered the highlighted debugging configuration

enter image description here

and then run the debugger - it will hit your breakpoints !!

enter image description here

the connect command should look something like this:

/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 -Xms256M -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/var/folders/yn/4k6t1qzn5kg3zwgbnf9qq_v40000gn/T/confluent.CYZjfRLm/connect/logs -Dlog4j.configuration=file:/Users/myuserid/confluent-5.3.1/bin/../etc/kafka/connect-log4j.properties -cp /Users/myuserid/confluent-5.3.1/share/java/kafka/*:/Users/myuserid/confluent-5.3.1/share/java/confluent-common/*:/Users/myuserid/confluent-5.3.1/share/java/kafka-serde-tools/*:/Users/myuserid/confluent-5.3.1/bin/../share/java/kafka/*:/Users/myuserid/confluent-5.3.1/bin/../support-metrics-client/build/dependant-libs-2.12.8/*:/Users/myuserid/confluent-5.3.1/bin/../support-metrics-client/build/libs/*:/usr/share/java/support-metrics-client/* org.apache.kafka.connect.cli.ConnectDistributed /var/folders/yn/4k6t1qzn5kg3zwgbnf9qq_v40000gn/T/confluent.CYZjfRLm/connect/connect.properties
user_70850470

Connector module is executed by the kafka connector framework. For debugging, we can use the standalone mode. we can configure IDE to use the ConnectStandalone main function as entry point.

  1. create debug configure as the following. Need remember to tick "Include dependencies with "Provided" scope if it is maven project enter image description here

  2. connector properties file need specify the connector class name "connector.class" for debugging enter image description here

  3. worker properties file can copied from kafka folder /usr/local/etc/kafka/connect-standalone.properties
user_17659810
Related