How to plot max-in-1h-window of rate-per-1-second of events in Grafana?

user_11214323 0
Viewed 925

We have a rate limit of 500 fetches per second from an external API. I want to track how close are we to this limit as the user base grows on a scale of weeks. I don't need the exact rate every second, I need the maximum 1-second-rate in a 1 hour window. The fetches are in bursts so the rate over 1 second can be much higher than the average rate over 1 hour or even 1 minute.

I have a Prometheus counter on the fetches and can use this query:

max_over_time(sum(rate(fetches_counter_total{namespace="$namespace"}[1m]))[1h:1m])

This does the max-in-window correctly but only on 1-minute-average-rate and not 1-second-rate. When I change the ending to [1s]))[1h:1s]) Grafana says "No Data" and no plot is shown.

I thought that maybe Prometheus is not fast enough so I added to the code a log every time a fetch happens and I can get the rate of logs from Loki using this query:

sum(count_over_time({namespace="$namespace"} |= "Fetch happened log text" [1s]))

This gives the 1-second-rate accurately but when I set the range to a week there are much less points than seconds in a week and refreshing shows a different subset of points each time. To get the max-in-window I tried this query:

max_over_time(sum(count_over_time({namespace="$namespace"} |= "Fetch happened log text" [1s]))[5m:1s])

But it throws an error: parse error at line 1, col 15: syntax error: unexpected SUM, expecting NUMBER or { or (

Now I'm stuck. From the documentation, seems that unwraping the logs might help but I don't have any labels and don't understand how to apply unwrap to this situation.

In summary, is it possible to get 1-second-rate from Prometheus or max_over_time of number of logs from Loki?

logql grafana-loki grafana prometheus promql
2 Answers

In prometheus rate() function will always return per-second rate. The amount of time you specify only means "look at that much time to calculate the per-second rate".

So:

   rate(metric[5m])

means roughly "take last 5 minutes of data for metric and calculate average per-second rate of events counted by metric in that time range".

Prometheus has to have at least two data points in the time range specified there to calculate rate, so if metric has a data point roughly every 1m then even [1m] might be too short (cause data points are close to the limits of time window that it looks in). Probably minimum time range used should be > 3*$scrape interval (if you want to handle random missing samples - if not then > 3*$scrape_interval )

From this I gather that your original query might be correct (assuming you have >1 data point in your metric per minute).

If you have a scrape interval for this data that is <1 second then this is rather unusual for Prometheus and I don't think it will work well. Review how often do you have your data points and adjust the time range used accordingly.

user_4742560

I don't know how to solve the task in Prometheus, but it can be easily solved in VictoriaMetrics with rollup_rate and label_match functions:

label_match(
  rollup_rate(fetches_counter_total[1h]),
  "rollup",
  "max",
)

See this article for more details.

user_2749370
Related