TL;DR : 10% of my Kafka Streams joins do not work while they should
Version : Kafka Streams 2.8
I have a topology, where, I build a referential from different sources, with some foreign-key joins. Then, I want to join the resulting referential, abstracted as a KTable, with my "main" event stream, also abstracted as a KTable.
My problem is that, 90% of my left join works like a charm : an event is produced in the output, as expected. But, 10% of them miss. Most of those joins are triggered by an event in the left KTable (since referential does not move often), and the associated entry in the right KTable seems not to be found while the key exists. I checked that by looking in the changelog topic of the RocksDB right store.
I have seen nothing in the logs for the moment.
Additional things I checked :
- Both input topics have same number of partitions
- Data is copartitioned as expected
- There have been no version change of any library in the lifecycle of the application
Additional information :
- The key of my messages is an avro schema containing two non optional strings.
- I have a KTable/KTable join because we want that a change in the referential produces an update on our "main" event stream.
- My two KTables are quite big (500M entries). I don't think this could change the result of the joins, but who knows...
I have no idea where I could find any more information on what could cause those joins not to happen, if anyone has any idea or tips on where I can find more information on what's going on, feel free to share :)