I'm pretty new to Prometheus and Grafana so this may be an easy question but I couldn't find any way to achieve what I want.
TL;DR: What is the best way to use Prometheus and/or Grafana to alert me when the number of jobs in the queue is stuck (only growing and not decreasing)?
I have a Prometheus server that I use for monitoring and I have a Grafana for visualization and alerting. I also have a job queue that I'm monitoring using a "number of jobs in the queue" Gauge in Prometheus. The number of jobs is constantly getting up and down. It can quickly jump high and it can go down as jobs are getting processed (this can happen very fast or slow).
The problem is sometimes this queue is getting stuck (the number is not decreasing and stays the same or only grows) for various reasons and I'd like to have an alert when this happens but I don't know how to do that.
Here's what I tried so far:
- Have a constant threshold like if the number is >= 100 for more than an hour that triggered the alert. That is far from perfect as it creates lots of false positives (queue is processing, use slow). Also, that was not catching if the queue is stuck at < 100.
- Comparing
NOW - [some time ago] > 0. This is not working because new jobs are getting added constantly and that also created a ton of false positives. - Doing a weird combination of
if "NOW - [10m ago] > 0"and "NOW - [1h ago] > 0" and "NOW - [2h ago] > 0". That helps a little bit but is too ugly and hard.
I feel like there should be an easier solution to have an alert only when the value is not decreasing?