I will have to create a Kafka producer application that exports a fairly high amount of data from a database with minimal transformation. Each export job would be triggered by a message from another Kafka topic. I was thinking of using the transactional API provided by Kafka to achieve exactly-once semantics, even if the application fails, and an export job has to be restarted. And doing the export itself with Spark in batch mode. However, I cannot find any clue in the docs if I can have that kind of control over Spark - Kafka interaction. Can I specify the commit id? Can I have control over when to initialize, finish the commit?
(I want to use Spark's batch processing capabilites.)