How can I get the task details (ID and Connector) within Kafka Connect Runtime

user_13028688 0
Viewed 579

I am trying to create a custom connector and I want to log the task details (Connector and Task ID) within the task runtime. In the Task start and poll methods I want to log the task number and connector name. Is there any way I can get the program runtime context which provides the task details (Connector and Task ID).

Please note that the connector may have multiple tasks running parallel (tasks.max configuration) so I want to associate specific messages with the specific task which created it.

apache-kafka-connect apache-kafka
2 Answers

Mapped Diagnostic Context (MDC) logging was added to Apache Kafka 2.3 and you can log this information by setting:

log4j.appender.stdout.layout.ConversionPattern=[%d] %p %X{connector.context}%m (%c:%L)%n

This gives you logs like this:

INFO [sink-elastic-orders-00|task-0] Using multi thread/connection supporting pooling connection manager (io.searchbox.client.JestClientFactory:223)
INFO [sink-elastic-orders-00|task-0] Using default GSON instance (io.searchbox.client.JestClientFactory:69)

ref: https://www.confluent.io/blog/kafka-connect-improvements-in-apache-kafka-2-3

user_3506130

A given Connector has a context field, however I am not seeing in the API how you would get details about metadata of the running task

user_23086830
Related