Prometheus recording rule to keep the max of (rate of counter)

user_1710154 0
Viewed 687

Iam facing one dillema. For performance reasons, I'm creating recording rules for my Nginx request/second metrics.

Original Query

sum(rate(nginx_http_request_total[5m]))

Recording Rule

    rules:
    - expr: sum(rate(nginx_http_requests_total[5m])) by (cache_status, host, env, status)
      record: job:nginx_http_requests_total:rate:sum:5m

In original query I can see that my max traffic is 6.6k but in recording rule, its 6.2k. That's 400 TPS difference.

This is the metric for last one week

Last 7 days

Question :

Is there any way to take the max of the original query and save it as recording rule. As it's TPS, I only care about the max, not the min.

prometheus devops kubernetes linux
1 Answers

I think having having 6% difference on value in some very short burst is pretty OK.

In your query your are getting (and recording) an average TPS during the last 5 minutes. There is no "max" being performed there. The value will change depending on the exact time of the query evaluation - possibly why you see difference between raw query and values stored by recording rule. Prometheus will extrapolate data some when executing functions like rate(). If you have last data point at time t, but running query at t+30s, then Prometheus will try to extrapolate value at t+30s (often this is noticed as a counter for discrete events will show fractional values)

You may want to use irate() function if you are really after peak values. It will use at each evaluation two most recent points to calculate most current increase as opposed to X minutes average that rate() provides.

user_4742560
Related