I want to dive into stream processing with Kafka and I need some help to get my head around some design principals which are currently not very clear to me.
1.) assuming I have some real-time stock price data. would you make one topic "price" keyed (and therefore partitioned) by the stock symbol? Or would you make one topic per symbol? in example what happens if I decide to produce (add) some more stock symbols including a full history later on? now my history (ordering in the log) in the topic "price" is a mess, right? On the other hand for each price series I want to calculate the returns later on and if they are on different topics I have to keep track of them and start new stream applications for every single symbol.
2.) having now different real-time prices and I need to join an arbitary number of them into one big record. in example join all sp500 symbols into one record. since I do not have a price of all sp500 symbols for the very same time but maybe pretty close. how can I join them using always the latest price if one is missing at this exact time?
3.) say I have solved the join use case and I pump the joined records of all sp500 stocks back into Kafka. what do I need to do if I have made a mistake and I forgot one symbol? obviously, i want to add it to the stream. now I kind of need to whipe the "sp500" log and rebuild it right? or is there some mechanism to reset the starting offset to a particular offset (the one where I have fixed the join)? also most likely I have other stream applications which are consuming from this topic. They also need to do some kind of reset/replay. is it probably a better idea to not store the sp500 topic but make it part of a long stream process? but then I will end up doing the same join potentially several times.
4.) maybe this should be 1. since this is part of my goal ^^ - however, how could I model a data flow like such:
produce prices -> calculate returns -> join several returns into a row vector -> calculate covariance (window of rowvectors) -> join covariance with returns
-> -> into a tuple (ret, cov)
I am not even sure if such a complicated use case is possible using today's stream processing.