My IoT device gets scraped by an external Prometheus agent, which uploads the data to Grafana Cloud.
It outputs two interesting metrics: uptime which is the number of CPU ticks since the device was started, and later last_update which is the current time value (which was retrieved via NTP).
What I'm looking for is a metric that tracks the time elapsed since the last update, so I can set up an alert when the device hasn't reported in for an hour. I thought I could do something like:
time() - last_update{job="charger"}
This works correctly, and shows there's ~2 seconds of skew between those two values. But there's no data for the instants when the data is missing, so I'm missing precisely what I'm interested in getting. It's during that time that the value should be rising, and eventually trip the alert threshold.
There's also the option of using up which does appear to have the value 0 when there is no data but I'm not sure how to make that work.
Any ideas on how to implement this query?
Other answers to questions like this (E.g.: Alert on missing series/data) focus on updating the scraping config, not updating the query.
Thanks,
-mk