I am using micrometer with prometheus and grafana for dashbaords. my prom query is : rate(my_counter[10m]) Now, My current date is 1-Jun-2020 and I select the past dates in time interval of grafana as 1-Feb-2020 to 4-Mar-2020. In this case, the rate function will use only 10 minutes of data w.r.t. 4-Mar-2020 23:59:59. Is my understanding correct, if no can someone please explain how does it work?
1 Answers
Prometheus calculates rate(my_counter[10m]) independently per each point on the graph. If Grafana draws a graph on a time range from 1-Feb-2020 to 4-Mar-2020, it sends the request to /api/v1/query_range with the corresponding query args: start=1-Feb-2020, end=4-Mar-2020. The step query arg is calculated depending on the horizontal pixel resolution of the graph and the selected time range, so the total number of returned datapoints is proportional to pixel_width of the graph. Usually Grafana sets step to an interval, which corresponds to around 1000 points per graph. If the lookbehind window specified in square brackets (e.g. [10m] in rate(my_counter[10m]) is smaller than step query arg, then some raw samples aren't taken into account when drawing the graph. If the lookbehind window is bigger than step query arg, then some raw samples are used multiple times for rate() calculations over adjacent points on the graph.
Grafana provides $__interval template variable, which equals to step value - see these docs for details. So just use $__interval instead of fixed lookbehind window if all the raw samples must be taken into account when building the graph on any time range:
rate(my_counter[$__interval])
Side note: the $__interval in square brackets can be omitted when using MetricsQL: rate(my_counter). See these docs for details.
Full disclosure: I'm the author of MetricsQL and VictoriaMetrics :)
Related