Prometheus+grafana: How to graph events per time slot

user_2444661 0
Viewed 1640

I have been struggling to have a graph in grafana which shows number of events per x units of time, like 1min,5min, 30secs etc. Just to give an idea what I am looking for, please see following image:

enter image description here

I want to have a graph which shows time slots(like for 5 min slots, 5.30-5.35, 5.35-5.40, so on....) on X axis and Y axis shows events per time slot.

I have configured a counter and I can use prometheus's increase function to find delta of events over a period of time(say 1m) as:

increase(mycounter[1m])

But the graph generated is not like what I want. It gives more of an average of the increase over time windows, not the exact increase in each of the time window.

Can this be achieved?

grafana prometheus
1 Answers

You will never get the exact increase in each of the time windows due to the way Prometheus rate and increase work.

By setting the min step in Grafana to whatever window step you like and the range selection in the query itself to $__interval and $__rate_interval once Grafana 7.2 is released, you get pretty close to what you want. But consider that once you zoom out far enough the window will start to grow if there are not enough pixels to fit all "windows". In addition having the min step and range exactly the same can and will lead to increases getting lost. $__rate_interval fixes that.

Edit:

The queries from Grafana to Prometheus contains a step parameter that Prometheus "steps through" along the given time range. For every step the respective query, for example rate(counter_total[5m]) is evaluated. The min step is a lower bound for Grafana. There are several other locations where you can define something similar. For example in your data source settings where you set your scrape interval.

The actual step value that is used in the end is calculated based on a number of other factors. For example the number of pixels available. But you can also fix the number of total steps you want, for example to 10. Here is where $__interval comes in. It basically synchronizes the step with the rate range and ensures that you have always the highest possible resolution while preventing under- and overfetching. Disadvantages: Graph may become quite spiky and noisy and you might prefer the look of a 5/10/15m rate. In addition, you can lose increases in your data if your rate range is exactly the same as the step. But the same is true with a fixed rate range and a big enough time range. $__rate_interval fixes that by always adding the scrape interval to the range selector.

Again, I recommend watching https://www.youtube.com/watch?v=09bR9kJczKM

user_73913310
Related