How to reprocess messages in Kafka Connect?

user_11291191 0
Viewed 78

I am working with a Kafka Sink Connector which reads from a Kafka topic and puts the data into a target database (in my case it is a Neo4j instance) .The messages need to be processed strictly sequentially since they are not idempotent. My question is if for some reason an exception occurs, for e.g. 1. Datbase goes down , 2. Connectivity to DB lost , 3. Schema parsing failure , how can we reprocess the message ?

I understand we can run with error.tolerance=none configuration and redirect failure message to a dead letter queue. But my question is there any way we can process a selected message again ? Also , is there any audit mechanism to track how many messages are processed, to seek from a given offset (without manual offset reset).

Below is my connector configuration . Also suggest if there are better data integration technologies apart from the kafka connectors to sink the data into a target database.

{
    "topics": "mytopic",
    "connector.class": "streams.kafka.connect.sink.Neo4jSinkConnector",
    "tasks.max":"1",
    "key.converter.schemas.enable":"true",
    "values.converter.schemas.enable":"true",
    "errors.retry.timeout": "-1",
    "errors.retry.delay.max.ms": "1000",
    "errors.tolerance": "none",
    "errors.deadletterqueue.topic.name": "deadletter-topic",
    "errors.deadletterqueue.topic.replication.factor":1,
    "errors.deadletterqueue.context.headers.enable":true,
    "key.converter":"org.apache.kafka.connect.storage.StringConverter",
    "key.converter.enhanced.avro.schema.support":true,
    "value.converter.enhanced.avro.schema.support":true,
    "value.converter":"io.confluent.connect.avro.AvroConverter",
    "value.converter.schema.registry.url":"https://schema-url/",
    "value.converter.basic.auth.credentials.source":"USER_INFO",
    "value.converter.basic.auth.user.info":"user:pass",
    "errors.log.enable": true,
    "schema.ignore":"false",
    "errors.log.include.messages": true,
    "neo4j.server.uri": "neo4j://my-ip:7687/neo4j",
    "neo4j.authentication.basic.username": "neo4j",
    "neo4j.authentication.basic.password": "neo4j",
    "neo4j.encryption.enabled": false,
    "neo4j.topic.cypher.mytopic": "MERGE (p:Loc_Con{name: event.geography.name})"
}
kafka-consumer-api apache-kafka-connect apache-kafka
1 Answers

For non fatal exceptions, the connector will write to a dead letter topic.

You'd need another connector or some other consumer to read that other topic to process that data. Since it's a topic, there's no straightforward way to "process a selected message"

JMX metrics or Neo4j database metrics should both be able to tell you approximately how many messages have been processed over time

user_23086830
Related