Trigger alert when prometheus metric goes from "doesn't exist" to "exists"

user_10441909 0
Viewed 527

I'd like to configure a prometheus alert to trigger when an existing metric shows up with new label values.

Example: a metric named my_metric has a label named sv. When my_metric{sv="john"} goes from "doesn't exist" to "exists" or "has value", it would trigger the alarm. However, I don't know in advance what the new value "john" will be.

Is this possible? What would the expression look like?

prometheus
1 Answers

In general, the query looks like this:

my_metric unless my_metric offset 10m

Simply put, the above gives you all my_metric time series, except those that were present 10m ago:

  • unless removes from the output time series with equal label sets;
  • offset 10m changes the evaluation time for the last my_metric from 'now' to 10m ago. In the context of your task, this would also define how long the alert will be firing before going resolved automatically.

Example:

# my_metric 10 minutes ago
my_metric{foo="bar"} 1.0

# my_metric now
my_metric{foo="bar"} 1.0
my_metric{foo="baz"} 1.0

# query result:
my_metric{foo="baz"} 1.0

Now, in the basic form above you will receive an alert for any new label set. If you want to watch for appearance of some specific labels, then add some aggregation:

avg by(label1, label2) (my_metric) unless avg by(label1, label2) (my_metric) offset 10m

Replace label1, label2 on both sides with label names that make sense in your case. You can also use other aggregation functions (min(), max(), sum(), etc) instead of avg().

user_113445020
Related