Prometheus count specific value occurrences

Viewed 1206

Can someone please help me to solve the task by counting a gauge value?

I have one metric but it always has different device labels, and it can only be 1 or 0, where 1 = error and 0 = OK.

I need to calculate the number of times when metrics were 1 by range variable provided in Grafana.

I tried operators sum, sum_over_time, and count and only sum_over_time provides truth result, but for some reason, it shows me one value multiple times

sum_over_time(aqa_device_health_checker{env="dev", device="FOO"}[1d])
1 Answers

I came up with two solutions to this, choose whichever suits you best. For the purpose of simplifying things, let's assume your Prometheus scrape at 15 seconds interval and the error state lasted for 1 minute. Then, the gathered data would look like this:

state_metric 0 @t
state_metric 1 @t+15s
state_metric 1 @t+30s
state_metric 1 @t+45s
state_metric 1 @t+60s
state_metric 0 @t+75s

With changes()

This shows how many state changes were there. It would return 1 for the exemplary data above and it only gives adequate results if the gauge in question can hold exactly two possible values (for example 1 and 0).

changes(state_metric[1d])/2

changes() shows how many times the metric value has changed during the interval, while division by 2 is to compensate the state change back to normal. This is the downside of this method, which makes it only usable for detecting quick changes of state. But you probably have an alert when the error state hangs for some time, so I think this shouldn't be really a problem.

With a subquery

This is more precisely what you asked:

the number of times when metrics were 1

But there is a catch: with the exemplary data above, the query below will return you 4:

sum_over_time(count(state_metric == 1)[1d:])

[1d:] means repeat that instant query (count(state_metric == 1)) for each data point during last 1d. This is precisely the number of times when state_metric was 1 and it can be useful, for example, to calculate the downtime (just multiply by the scrape interval). Unlike the first method, this can work with any number of possible states, since you can define what you need in the condition.

Related