To learn Kafka, I create a microservices application: A trading system that can be used by GUIs or trading bots. Some microservices are responsible for getting market data from different brokers, and "produce" the market data into Kafka topics. A Trading bot or a GUI can then consume these market Data on the topics. For this use-case, Kafka is the PERFECT tool!
But, What if the trading bots want to consume some metrics on the real-time data. What is the best solution to produce the result for computed metrics (Windowed, or not)?
For instance, Trading-Bot-1 wants the real-time moving average on BTCUSD price for the last 200 minutes, and Trading-Bot-2 wants the same information and the variance of the ETHBTC price from the time I have opened a position to now.
I have the feeling that Kafka is not a good choice. During my research, I have seen it is a bad practice to consider Kafka as a way to create a wire between two micro-service. Throughout my readings, I got the feeling that Kafka is not suitable for "request-style" messages, which is not purely an event as we expect a result.
The dirtiest solution would be to create Topic-1 to send metrics request (i.e, "Request XXX, Need moving average on 150 last prices for BTCEUR"), and Topic-2 to send the result ("For the request XXX, result is YYY").... Very inefficient I suppose.
Another solution would be to create a consumer group on the market data topic, and the consumers of this group will be the services responsible for producing the metrics into a kafka topic named "Metrics". Not satisfying also, it computes metrics that may never be used by the current consumers of the system, and it also limits me to a set of metrics predefined into the metrics services, so Trading Bots and GUIs won't be able to request metrics with custom inputs (i.e, computing in real-time the present value of a future with a custom risk-free rate).
Please note that I would like these metrics to be provided by my services, I don't want the consumers of the system (GUIs or trading bots) to be responsible for computing the metrics themself.
What do you guys think? Is Kafka suitable for metrics computation "in real-time" or should I consider another solution?
Many thanks!