Why is the moving average higher than the actual series in prometheus

By default Prometheus wraps time series selectors into last_over_time() rollup function with 5 minutes lookbehind window in square brackets if the time series selector isn't wrapped into any rollup function. So the sum(gin_in_flight_requests{app="my-service",cluster="prod", url="/api/v1/url1"}) query is automatically converted into the following query before execution:

sum(
  last_over_time(
    gin_in_flight_requests{app="my-service",cluster="prod", url="/api/v1/url1"}[5m]
  )
)

See these docs for more details.

E.g. this query takes into account a subset of raw samples, actually the last raw samples just before each point displayed on the graph. It ignores the remaining raw samples. So it may return values smaller than the sum(avg_over_time(...)) query. If you want taking into account all the max raw samples, then use max_over_time function.

P.S. If you want capturing all the raw sample maximums and minimums on the selected time range in Grafana, then just use max_over_time() and min_over_time() queries with $__interval lookbehind window in square brackets:

sum(max_over_time(...[$__interval]))

and

sum(min_over_time(...[$__interval]))

P.P.S. FYI, an alternative Prometheus-like monitoring solution I work on - VictoriaMetrics - provides a rollup function, which simultaneously returns min, max and avg values on the selected time range. E.g. it can be used instead of three queries with min_over_time(), max_over_time() and avg_over_time() functions.

The yellow line are interpolated query evaluation points. Not the raw samples in the database.

See Average value not calculated right? and How does Prometheus DB calculate average value.

Interpolation can get you higher or lower values than the original data.

Example

Raw data:

• data point value 1 at second 1
• data point value 10 at second 10
• data point value 1 at second 20

Panel:

• interpolated value at second 5 is round about 5
• interpolated value at second 15 is round about 5

But the average value between second 5 and second 15 is 10 (just a single data point in this interval, no interpolation here).