Aggregating counter metric from a Prometheus exporter that doesn't respect monotonicity

user_11660036 0
Viewed 26

We have a system with a Prometheus exporter. It exposes a counter metric. However, the exporter sometimes misbehaves and breaks monotonicity like this:

  • T1: X
  • T2: X + 1
  • T3: X

Example:

Graph of counter misbehaving

As a result, when applying rate() or increase() on the counter, we get huge spikes, because Prometheus understands that the counter was reset to 0, and then increased again. So we get increases series like:

  • T1: 0
  • T2: 1
  • T3: X

X can be in the thousands, so this completely messes with the aggregated metrics.

Example of increase(metric[3d]):

Graph of increase() on counter that misbehaves

Do you have any idea of how could we work around this wrong values with native PromQL? For simplicity, let's assume that the errors are always in the form of X/X+1/X.

I've thought of using delta() as if this metric were a gauge, but we can't do that, because in some scenarios it actually resets to 0 as counters normally do, and delta() won't understand it.

Fixing the exporter is out of scope.

prometheus promql
1 Answers

increase() function in Prometheus doesn't provide the ability to gracefully handle small spikes in counter metrics. Possible workaround is to use delta() function instead of increase(), and filter out negative values when the counter resets or goes down after the spike:

delta(metric[5m]) > 0

But this method doesn't work properly for long lookbehind windows in square brackets. In this case the query can return long gaps and too small values on long time ranges after counter reset.

Fortunately, there is an alternative Prometheus-like monitoring solution, which gracefully handles cases with counters with possible spikes - VictoriaMetrics (I'm the core developer of VictoriaMetrics). Its increase and rate functions handle such spikes in an expected way by ignoring small counter decreases.

Additionally, VictoriaMetrics provides remove_resets function, which can be used for removing resets from counter metrics.

user_2749370
Related