We have a few thousand IoT devices that send us their temperature every second. The input source can be MQTT or JSON (or a queue if needed).
Our goal is to near continuously process data for each of these devices and calculate the average for the last hour, etc. Lastly, we want it to send a notification if it exceeds a certain level.
What are some good alternatives stream processing tools that are flexible and scalable? Is Kafka the way to go, or are there alternatives?
Have you looked at IBM's IoT foundation?
https://www.ibm.com/cloud/internet-of-things
IBM also offers kafka based Event Streams. https://www.ibm.com/cloud/event-streams
You could use a new platform called Quix. https://quix.ai/
They have an MQTT connector - hiveMQ I believe. They also have connectors for slack or twillio to send your notification. You just have to write your logic for whatever calculation you want. although they do provide some pre built tansformations for thing slike rolling average etc.
I believe its running on Kafka but you dont have to know the ins and outs of setting all of that up.
I built something similar on there and it works really well.
