Alert manager triggers web-hook repeatedly for same alert

user_2816429 0
Viewed 372

I have configured the alert manager rule to trigger alert when Prometheus metric changes from 0 to 1 It triggers a webhook alert upon metric changed from 0 to 1 But alert manager keeps triggering webhook, duplicate alerts for the same metric change.

Is there a config to prevent silencing further alerts from the alert manager?

below is my alertmanager config

global:
  resolve_timeout: 15m
  http_config: {}
  smtp_hello: localhost
  smtp_require_tls: true
route:
  receiver: web.hook
  group_by:
  - ccu_code
receivers:
- name: web.hook
  webhook_configs:
  - send_resolved: true
    http_config: {}
    url: http://service:8080/alarms
    max_alerts: 0
templates: []
prometheus-alertmanager prometheus
1 Answers

You get these repeat alerts because AlertManager has this setting:

# How long to wait before sending a notification again if it has already
# been sent successfully for an alert. (Usually ~3h or more).
[ repeat_interval: <duration> | default = 4h ]

https://prometheus.io/docs/alerting/latest/configuration/

I think the reason it's designed this way is that you want to keep getting notification about an active issue, in cases where you forgot to handle it, etc.

user_10503890
Related