I am using Kafka with .Net Core and I have 2 topics. In which both contain information from the same entity and are partitioned by the same key.
On my application, I´d like to Join both informations to process them together. This process is in-memory as it requires a reasonably low latency.
My application has multiple nodes running, so when one node spins up, It is assigned some partitions from the first topic T1 (say T1:P1, T1:P3) and some partitions from the other topic T2 (say T2:P2, T2:P4)
My issue is that I cant guarantee that a single application node will be assigned partitions of the same entity from both topics. It may be that a given entity ended up in T1:P1 for the first topic and T2:P3 on the second topic. In the example above I would not be able to mix them together.
I known that I can use a consistent partitioner on the producer side, so that at least I known that my entity ended up on the same partition number on both topics (e.g. T1:P1, T2:P1). It still doesn´t help me since I can be assigned any partition.
It seems that Kafka Streams would be suited for this scenario, however it still doesn´t have an implementation for .net.
There are some possible solutions I thought of:
- On the consumer, let kafka client assign partitions of
T1, and in the callback ofPartitionsAssigned, manually assign the same partitions onT2. - Make service stateless, and use a database. Save/retrieve the state upon each new message.
- On my particular case,
T1has a very high frequency andT2a lot less. I was thinking about reading ALL partitions fromT2on every node (e.g. use a random groupId).
But all solutions present problems:
- On strategy (1), I am wary to loose messages of
T2during a partition rebalance. - Strategy (2) will introduce a latency, that I really didn´t want. Even if using redis for example it might become a problem.
- Solution (3) simply doens´t scale. I might be able to do it this time, but it will put some memory pressure on all my nodes.
Are there any other solutions I didn´t think of ?