Prometheus Alertmanager Expression Problem/Question(s) concerning Lists

user_7199628 0
Viewed 387

I am trying to create an prometheus-alert-expression that checks if a mountpoint is mounted and sends an alert if the mountpoint is missing .. The Idea was something like this:

groups:
  - name: mountpoints 
    rules:
      - alert: /ghome missing 
        expr: absent(node_filesystem_avail_bytes{mountpoint="/ghome", instance="my.machine.org:9100"}) 
        for: 60s
        labels:
          severity: critical
        annotations:
          summary: "/ghome missing on ({{ $labels.instance }})."
          description: "VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

This kinda works. But is there a way of passing a list/vector to the mountpoint(s) and/or instance(s).

Using this Expression I'll have to write an alert-rule for each instance and each mountpoint.

I had the Idea of trying regular-Expressions like

    expr: absent(node_filesystem_avail_bytes{mountpoint=~"/ghome|/something|/other", instance=~"my.machine.org:9100|another.machine.org:9100"})

.. but this obviously does not work.

Does anybody have an idea how to implement this ?

Greetings

Volker

alertmanager prometheus-alertmanager expression
1 Answers

When we pass an expression such as absent(my_metric{label=~"1|2"}), this is evaluated as such: my_metric{label=~"1|2"} might return 4 possible results:

  1. No result
  2. my_metric{label="1"}
  3. my_metric{label="2"}
  4. both my_metric{label="1"} and my_metric{label="2"}

And the absent function is then called upon these results, and for absent to return "1" it will only do so when there are no results. Missing the case when 1 of them is absent.

Unfortunately there's no one-liner for this, we'll have to be explict with absent, we can either have multiple alert-rules or use the or operator such as:

absent(my_metric{label="1"}) or absnet(my_metrci{label="2"})

user_10503890
Related