Prometheus - calculating the percentage increase

user_1659806 0
Viewed 648

I'm very new to using Prometheus metrics and I've been asked to write an alert based on whether a value has increased by a 10% over the last five minutes.

To be more concise, I've to report when my Kafka topic deadletters count increases by 10% within 5 minutes.

I can count the current deadletters on the topic/sub using this query:

count by(topic) (kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"})

What I now need to do is get the same number from 5 mins ago and then calculate the percentage increase from then and now:

Percentage Increase = [ (Final Value - Starting Value) / |Starting Value| ] × 100

Can anyone give me a steer on how to calculate the percentage difference?

I've seen examples like this but cant get it to work for me: (sum by(topic) (increase(kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+"}[5m])) > 0)

prometheus apache-kafka metrics
1 Answers

If you use the following query for calculating the value at the moment:

count by(topic) (kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"})

then the value 5 minutes ago can be calculated by adding offset 5m to the query:

count by(topic) (kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"} offset 5m)

Note that the offset modifier must be put immediately after closing curly brace in the query above. See these docs for more details on this.

Given that, the following query would show the percentage increase for the value during the last 5 minutes:

(
  count by (topic) (
    kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"}
  ) - count by (topic) (
    kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"} offset 5m
  )
) / count by (topic) (
  kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"} offset 5m
) * 100

P.S. Note that the query contains multiple repeated parts. These parts can be simplified to the following MetricsQL query using with templates from VictoriaMetrics:

with (
  q = count by (topic) (
    kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"}
  )
)
(q - q offset 5m) / (q offset 5m) * 100
user_2749370
Related