Is there any way to reduce this code of prometheus alert expressions? I have multiple similar expression only the source instance is different

user_5445436 0
Viewed 159

Suppose I am getting the metrics from a service in event_processing_bucket tag where instance are like source=ONE, source=TWO, source=THREE ...... TEN

Currently I am using the following way to get the alert, but here I have written a separate expression just because i have to get data for every single source.

Is there any way to reduce this duplicate code. so that i could write only one alert rule and it will alert for all separately based on its respective value

Here are the prometheus alert expressions,

- alert: ONE_SLA_GREATER_THAN_5DAYS
  expr: sum(rate(event_processing_bucket{source="ONE"}[1m])) > 5
  for: 1m
  labels:
    severity: warning
    team: mySlackChannel
  annotations:
    description: ONE_SLA is GREATER_THAN_5DAYS
    summary: ONE_SLA is GREATER_THAN_5DAYS
- alert: TWO_SLA_GREATER_THAN_5DAYS
  expr: sum(rate(event_processing_bucket{source="TWO"}[1m])) > 5
  for: 1m
  labels:
    severity: warning
    team: mySlackChannel
  annotations:
    description: TWO_SLA is GREATER_THAN_5DAYS
    summary: TWO_SLA is GREATER_THAN_5DAYS
.
.
.

- alert: TEN_SLA_GREATER_THAN_5DAYS
  expr: sum(rate(event_processing_bucket{source="TEN"}[1m])) > 5
  for: 1m
  labels:
    severity: warning
    team: mySlackChannel
  annotations:
    description: TEN_SLA is GREATER_THAN_5DAYS
    summary: TEN_SLA is GREATER_THAN_5DAYS

Please guide me to write single expression code if possible. if not please specify.

Thanks in advance!!

prometheus-alertmanager promql
1 Answers

One way is to group by

histogram_quantile(0.95, sum(increase(event_bucket[5m])) by (le, source)) > 5

later result values can be used to trigger those many alerts

user_54454360
Related