We get metrics about the number of messages in queues from our instance of ActiveMQ Artemis 2.10.0 to Prometheus, and I need to be notified when for a certain amount of time (let's say 8 hours) the queue grows and does not decrease (usually this indicates a problem with the service that pulls messages from queues).
Like this:
But if I see something like this in the image below i.e. peak growth followed by a decrease, then the alert should not be triggered:
Now I use this expression, but sometimes it does not work correctly due to large growth spurts even with a subsequent decrease:
floor((predict_linear(artemis_message_count{job="activemq",queue=~".*"}[24h], 3600 * 24 * 1))) - max_over_time(artemis_message_count{job="activemq",queue=~".*"}[24h]) > 0
Can't figure out which expression is better to use in order to have fewer fake alerts. Would be grateful for a hint.