Prometheus (metric) relabel config with inverse regex match / negative lookahead

user_4391861 0
Viewed 2527

Right now I'm scraping metrics from a Node Exporter. Some of the metrics which the Node Exporters exports have a mountpoint label.

I'd like to drop time series that have this label and do not match a regular expression. I tried using the keep action (as I'd like to keep time series that do match this regular expression) but this also drops all other metrics that do not have the mountpoint label.

metric_relabel_configs:
  - source_labels: ['mountpoint']
    regex: '(\/home|\/var\/domains)\/something.*'
    action: keep

I tried using the drop action too but this requires me to inverse the regular expression using a negative-lookahead (which isn't supported because Prometheus is written in Go of course).

What are my options in this?

Important, I do not have control over the way the Node Exporter is configured, thus I can't configure the Node Exporter itself to not export metrics for some specific mountpoint (if this is even possible).

prometheus-node-exporter prometheus
2 Answers

If I understand you correctly, then the following should meet your needs. I tested it in the Relabeler online tool and seems to do what you want?

Using these relabeling rules:

- source_labels: ['mountpoint']
  regex: '(\/home|\/var\/domains)\/something.*'
  target_label: __tmp_keep_me
  replacement: true
- source_labels: [__tmp_keep_me]
  regex: true
  action: keep

The following (example) object labels would be kept:

mountpoint: "/home/something/"
job: "node"
fstype: "ext4"

While these would be dropped:

mountpoint: "/tmp"
job: "node"
fstype: "ext4"

Note that I used a tip Brian Brazil shared in the Or in relabelling article and adapted it.

user_3965670

Just add | to the end of the regex, so it matches time series without mountpoint label:

metric_relabel_configs:
- source_labels: [mountpoint]
  regex: '(/home|/var/domains)/something.*|'
  action: keep

Shameless plug (I'm the author of vmagent): vmagent supports easier-to-understand-and-use construct for this particular case with if option:

metric_relabel_configs:
- if: '{mountpoint=~"/home|/var/domains)/something.*|"}'
  action: keep

The if option may contain arbitrary time series selector. See these docs for more details.

user_2749370
Related