How to calculate the average value in a Prometheus query from Grafana

user_7107177 0
Viewed 32417

I was trying to create a Prometheus graph on Grafana, but i can't find the function which calculate the average value.

For example , to create a graph for read_latency, the result contain many tags. If there are 3 machine, there will be 3 tag seperately, for machine1, machine2, machine3. Here is a graph(click to show) Prometheus

I want to combine these three together, so there will be only one tag : machines, and the value is the average of those three.

It seems that Prometheus query function doesn't have something like average(), so I am not sure how to do this.

I used to work on InfluxDB, and the graph can show like (click to show): influxDB

grafana prometheus
3 Answers

Use built-in $__interval variable, where node, name are custom labels (depending on you metrics):

sum(avg_over_time(some_metric[$__interval])) by (node, name)

or fixed value like 1m,1h etc:

sum(avg_over_time(some_metric[1m])) by (node, name)

You can filter using Grafana variables:

sum(avg_over_time(some_metric{cluster=~"$cluster"}[1m])) by (node, name)
user_3348310

Short answer: use avg() function to return the average value across multiple time series. For example, avg(metric) returns the average value for time series with metric name.

Long answer: Prometheus provides two functions for calculating the average:

  • avg_over_time calculates the average over raw sample stored in the database on the lookbehind window specified in square brackets. The average is calculated independently per each matching time series. For example, avg_over_time(metric[1h]) calculates average values for raw samples over the last hour per each time series with metric name.
  • avg calculates the average over multiple time series. The average is calculated independently per each point on the graph.

If you need to calculate the average over raw samples across all the time series, which match the given selector, per each time bucket, e.g.:

SELECT
  time_bucket('5 minutes', timestamp) AS t,
  avg(value)
FROM table
GROUP BY t

Then the following PromQL query must be used:

sum(sum_over_time(metric[$__interval])) / sum(count_over_time(metric[$__interval]))

Do not use avg(avg_over_time(metric[$__interval])), since it returns average of averages, which isn't equal to real average. See this explanation for details.

user_2749370
Related