Kafka Connect Alerting Options?

(I still can't comment so to respond to clay's answer...)

NOTE: There is a bug in the JMX metrics for task/connector status (at time of posting: 5/11/2020)

1) When a task fails, it's status metrics dissapear. This is a known issue and there is a fix in progress. A Jira can be found here and PR can be found here.

2) Don't use the Connector metric to monitor the status of the tasks. The Connector can show up as running fine but the tasks can be in a failure state, you need to monitor the tasks directly. This is mentioned in Confluent's Connector monitoring tips where it says:

In most cases, connector and task states will match, though they may be different for short periods of time when changes are occurring or if tasks have failed. For example, when a connector is first started, there may be a noticeable delay before the connector and its tasks have all transitioned to the RUNNING state. States will also diverge when tasks fail since Connect does not automatically restart failed tasks.

I know that this is a really old question, so when we ran into a similar issue as we use Kafka Connect really heavily, and as its very difficult to individually monitor each connectors especially when you are looking at managing more than 150+ connectors.

Hence we have written a small Kotlin based application, which accepts a config.json where you can specify the cluster config and if smtp config is specified, it will keep on polling the cluster based on a specified recursion interval specified and will send mail based alerts.

If it fits your use-case, please do use and do raise issues in-case you face any.

The link to the repo is as follows: https://github.com/gunjdesai/kafka-connect-monit

The image is also pushed on Docker Hub and you run it directly using the following command.

docker run -d -v <location-of-your-config-file.json>:/home/code/config.json gunjdesai/kafka-connect-monit

Hope this maybe helpful to you

Since this post was written/answered, Kafka Connect began providing its own official metrics. The Apache Kafka Connect provides metrics in legacy JMX format.

If you use the Confluent Kafka Connect Helm Charts (https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-kafka-connect), they include a Prometheus metrics exporter.

I monitor and alert on cp_kafka_connect_connect_connector_metrics{status="running"} from the Confluent Helm Chart Prometheus chart, but there are many variations to that.

Using the official Kafka Connect metrics is generally preferable for any automated monitoring + alerting setup. This option wasn't available back when this post was written + answered.

FYI, Kafka still doesn't expose lag metrics, so you still need third party options to monitor and alert on lag.

I know it's a bit late, but this might complete what people suggested here, one way to improve your KC cluster monitoring would be to use this Kafka Connect REST extension : https://github.com/LoObp4ck/kafka-connect-healthchecks

And then have your monitoring periodic job to check for this endpoint to ensure all connectors tasks are running fine. We use it in production and it does the job

The jar is also available in maven central as follows :

<dependency>
  <groupId>net.loobpack.kafka-connect-healthchecks</groupId>
  <artifactId>kafka-connect-healthcheck-extension</artifactId>
  <version>1.0.0</version>
</dependency>