We are building a exception management tool with kafka. There will be source connector - which will pull records from physical file. On the other hand, there will be sink connect (mongodb-sinkconnect) which will pull the records from the topic and push it to mongoDb. Everything works fine.
We need to capture the event (for audit purpose) in a different topic. Events such as,
- Source Task (File polling task) start event Example, if file A received
- Source Task (File polling task) end event Example, if file A fully processed
- Sink Task (pushing records to mongodb task) start event Example, file A's records started processing by mongodb-connect
- Sink Task (pushing records to mongodb task) end event Example, file A's records fully pushed to MongoDB
I have a couple of questions here: 1.We are able to send events to different topic by instantiating a KafkaProducer inside SourceTask and once file is completely processed, we send an event
public class FileSourceTask extends SourceTask {
private Producer<Key, Event> auditProducer;
public void start(Map<String, String> props) {
auditProducer = new KafkaProducer<Key, Event>(auditProps);
}
public List<SourceRecord> poll() {
List<SourceRecord> results = this.filePoller.poll();
if(results.isEmpty() && eventNotSentForCurrentFile) {
Event event = new Event();
auditProducer.send(
new ProducerRecord<Key, Event>(this.props.get("event.topic"), key, event));
}
// futher processing
}
Is the above approach coorect?
- The above solution works fine - as it runs with one task (maxTasks = 1), but for our use case, achieving this is very difficult in sink task (mongoDB connect). Since this topic is partitioned, there will be many tasks created. We are unable to track the start event and end event of sink task
Please suggest an approach to solve this.
Thank you so much.