Setup prometheus alerts for different environments

user_1801334 0
Viewed 357

I would like to setup more or less the same alert for prod and dev( for example: high cpu usage) I.e. i woud like to see [prod] my alert and [dev] my alert. I can define separate alert for prod and separate alert for dev. it will work ok. But will be good to have something like template of alert and change title and instance only for prod and dev. Is it possible?

prometheus
1 Answers

Often you can do with one alert:

  1. Use or statements:
groups:
- name: alerts
  rules:
  - alert: MyAlert
    expr: my_metric{env="dev"} < 10 or my_metric{env="prod"} < 1
  1. Define thresholds as metrics:
groups:
- name: thresholds
  interval: 1h # you do not need to evaluate these often
  rules:
  - record: my_metric:alert_threshold
    expr: 1
    labels:
      env: prod
  - record: my_metric:alert_threshold
    expr: 10
    labels:
      env: dev

- name: alerts
  rules:
  - alert: MyAlert
    expr: my_metric < my_metric:alert_threshold

Label and annotation values can be templated using golang template syntax (some examples). But, as far as I know, there is nothing (except for third-party tools) to template for value. For example, I use Ansible along with jinja2 templates to render some rules in a loop and deploy Prometheus configuration.

user_113445020
Related