I'm working on a solution for a real-time processing use case. The current implementation involves two different Kafka Streams applications and a middle Kafka topic. I'm using Spring Cloud Stream Kafka Streams Binder and the DSL.
input-topics --> [PRE-PROCESSOR] --> middle-topic --> [PROCESSOR] --> output-topic
Despite the simple code, the generated topology for the first application is rather complex since it uses a foreign-key KTable-Ktable join. The second application is non-trivial too, handling different use-cases and tombstones generation. Each topology has ~20 processing nodes and ~3/4 materialized state stores.
The solution is fully functional but doesn't perform very well in my opinion: the main concern is about latency. The application doesn't (yet) any significant computation to be responsible for the processing slowdown. Tinkering with different configurations as suggested here (mainly linger.ms for kafka streams producers) has improved the situation but even at a very low load (10 messages/second), we observe an end-to-end value of ~400ms, with spikes of 800/1200ms for the first result in the batch. At higher load (20-30 messages/second), the solution simply strives to catch up with high CPU processing and broker load (hence a throughput issue).
Is this the expected result for a non-trivial Kafka Stream application or I'm missing something? I understand that the middle topic and the internal repartitioning topic generated by the DSL lead to a producer-topic-consumer chain and a detrimental effect, but I was expecting something in the range of 50-100ms on low load for a millisecond processing latency library.
Thank You