I want to track if any application had stopped consuming from kafka topics. For that, I added Kafka Consumer lag alert rule in alert manager which sends alerts on slack channel whenever condition meet.
I am doing, group by (consumergroup) and sum the metric and if value > 0 then send a alert. This is working fine but due to high volume and low volume topics, alerts flooded in channel.
I want to add alert rule which can handle both high volume and low volume topics Kafka Consumer lag.
I also took the reference from - https://gist.github.com/Andor/18fe55fd6e08ec130988ef373473185e#file-kafka-lag-yaml-L28 but value is hardcoded. Which cannot work in case of low volume Kafka topics.
What will the rule which can handle both high volume and low volume topics Kafka Consumer lag?