I'm about to implement a streaming infrastructure for my organization based on Kafka and Spark. However i am puzzled at deciding the best way to go when it comes to ingesting data in Kafka.
Many solutions are indeed possible for that task
- Spark itself could be use to read from external source and write to Kafka. I don't want to use that road.
- Kafka Connect
- Kafka Client API (Producer and consumer)
- Akka-Stream Kafka (Which could be though of as Reactive Kafka client as far as I understood but i am not sure)
In order to make my choice, of course i could go on a journey of trying everything out on my own, however, i wonder if anyone already went trough that hurdle.
I am leaning toward (4). Hence to whomever has had some experience with those frameworks for the task at end, i wonder if he could share with me the experience.
In particular, I would like to know the pros and cons observed between using (4) and (2). What makes Kafka Connect a better choice for ingestion. Is it really much more work to use (4). Is Kafka Connect, reactive ? Does Kafka connect handle Back Pressure ?