I would like to calculate the percentage time that a given metric is non-zero in a time range. I know I can get the number of values in that time range using
count_over_time(my_metric[1m])
but what I would like is something like
count_over_time(my_metric[1m] != 0) / count_over_time(my_metric)
I can't do this because binary expression must contain only scalar and instant vector types.
Is there a way to do what I'm trying?
As of Prometheus 2.7, you can do this using a subquery.
Specifically, the following query will tell you what % of the time your query was nonzero, using a trailing 1 hour average.
avg_over_time((my_metric[1m] != bool 0)[1h:])
Note that the linked tutorial recommends that you still use a recording rule "to avoid re-computing the entire hour worth of subquery output at every evaluation interval."
While Prometheus provides subqueries, which can be used for estimating the proportion of samples, which aren't equal to zero, this doesn't give the exact value. For example, the following query gives the estimated proportion of non-zero my_metric values over the last hour:
avg_over_time((my_metric !=bool 0)[1h:30s])
But the estimation may be incorrect if the step value in square brackets after the : (30s in the example above) is bigger than the interval between raw samples stored in Prometheus (aka scrape_interval). In this case only a part of raw samples are taken into account when calculating the estimation. If the step is smaller than the scrape_interval, then some raw samples may be counted multiple times.
This issue is fixed in MetricsQL with count_ne_over_time function. For example, the following query returns the real proportion of non-zero my_metric samples over the last hour:
count_ne_over_time(my_metric[1h], 0) / count_over_time(my_metric[1h])
If my_metric samples are always non-negative, then the query can be simplified further with share_gt_over_time:
share_gt_over_time(my_metric[1h], 0)