I've been struggling (or thinking about) how can we measure, how long given alert is in 'firing' state. For example, let's use a basic monitoring rule from https://awesome-prometheus-alerts.grep.to/rules#host-and-hardware, noting complicated, just alert that informs us that filesystem is running out of free space:
- alert: HostOutOfDiskSpace
expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
for: 2m
labels:
severity: warning
annotations:
summary: Host out of disk space (instance {{ $labels.instance }})
description: "Disk is almost full (< 10% left)\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
How can I add a label, that will tell me the increment of how long this alert is occurring. I expect that once it fires, it will record the first occurrence at 0 minutes, and it will add up time and display different value after one minute. I was tinkering with recording rules, counters, avg's, but ended up in showing one value that doesn't change at all. Any ideas if this is even possible?