We are using Kafka in over a year now and want to move on with a deeper integration. But there is one concept I struggle with. I will try to explain what we want to achieve and how the solution looks like we came up with. The issue from my perspective, it’s not really an elegant way and that’s why I’m questioning if I got it right.
The Problem
We have one stream with complex structured events (nested structure). A consumer is taking those events, ripping them apart and is putting those pieces in separate topics. Behind each topic are a lot of different other service who enrich the events flowing through the topics in a stream processing fashion. At the end we have a number of topics, each with partial enriched events and we want to move them all together to bring them back as one complete event at the end of the entire process. But this easier said than done.
The Solution
At the end we have one service that consumes all the topics and builds a buffer until all the partial events flown in and puts them together to publish it in a new topic. The challenge for is to make sure we only produce events in the final topics that are complete. This works but has some pitfalls
- the buffer can not be internal, it has to be something external that multiple consumer can share informations
- we can theoretically run in timing issues and create dead entries
- we can have consistency issues
- and so on
My Question
Even if it works and I don’t think this is a very elegant way, are we on the right track or did we misunderstood something in the concepts and handling of Kafka events and stream processing? Are there any better ways to do it? Has somebody experience with this and can share some leanings or ways to integrate it in stable way?
Thanks! Any comments are very appreciated.