I'm currently reading into Kafka, trying to find a way to seperate our timeseries database storage engine from our application by making this more of a generic stand-alone microservice rather than an integral part of our application as it currently is.
We currently store our sample data (with timestamp) in a our in-house developed timeseries database and our application enables us to do a scala of analyses dedicated for our industry.
Kafka seems to be ideal for continously streaming data into it and out of it (what we need as well), but querying a datasource over a set period of time in the past, to get a data result stream, which therefore has a begin and an end, seems not to be a part of the scope of Kafka. That is, I can't find a proper way to create that in Kafka yet.
Having read this: https://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams/ I think I'm very close to what I want but I can't see yet how Kafka handles various queries for various recorded sample sets over different periods of time.
We have a lot of sample data sets over a long period of time (3+ years of 10000s of sample sets at a sampling rate of every 5 seconds to every 1 minute), and as our storage is limited, I hope Kafka does offer a more 'transient' way, than storing the result data of every request for 2 days (as it is set as default), if I understand it correctly, to get our data every time we want to do analysis.
I'm just so close, but I can't get my head around it how to do this properly in Kafka.
Thank you very very much for your time.