Prometheus - count how long alert is in firing state

Viewed 9

I've been struggling (or thinking about) how can we measure, how long given alert is in 'firing' state. For example, let's use a basic monitoring rule from https://awesome-prometheus-alerts.grep.to/rules#host-and-hardware, noting complicated, just alert that informs us that filesystem is running out of free space:

  - alert: HostOutOfDiskSpace
    expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of disk space (instance {{ $labels.instance }})
      description: "Disk is almost full (< 10% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

How can I add a label, that will tell me the increment of how long this alert is occurring. I expect that once it fires, it will record the first occurrence at 0 minutes, and it will add up time and display different value after one minute. I was tinkering with recording rules, counters, avg's, but ended up in showing one value that doesn't change at all. Any ideas if this is even possible?

0 Answers
Related