I have a Grafana graph with data coming in from Prometheus. I'm trying to get an alert set up so that if the change over one minute (or so, I'm flexible) is over 60, an email notification goes out. I can't seem to find anything in the documentation that suggests a way to do this, or if it's possible at all. Any ideas?
3 Answers
You need to use delta function. For example, the following query returns non-empty result (aka alert) when the metric foo increased by more than 60 during the last minute (see [1m] in square brackets):
delta(foo[1m]) > 60
It is possible to use any supported time duration in square brackets (aka lookbehind window) for caclulating the metric delta on different intervals. See these docs.
Related