prometheus relabel_config drop action not working

user_2067143 0
Viewed 990

I've been trying to drop unused metrics (before scrape) to lighten the load on a Prometheus cluster via relabel_configs:

- job_name: nginx-ingress-controller-metrics
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - <some-namespace>
  relabel_configs:
  # single
  - source_labels: [__name__]
    regex: nginx_ingress_controller_response_size_bucket
    action: drop
  # multiple
  - source_labels: [__name__]
    regex: nginx_ingress_controller_(request_size_bucket|response_duration_seconds_bucket|bytes_sent_bucket|bytes_sent_sum)
    action: drop

However, I am not seeing any of these metrics dropped (before scrape) and they are scraped and ingested as I can query them via PromQL.

I've tried many variations of the same config (i.e. quotes, parentheses, etc.). Any idea why these targets are still scraped/ingested?

prometheus
1 Answers

You cannot drop by __name__ before scraping (i.e. in relabel_configs), this label does not yet exist. relabel_configs happens before establishing a connection to the exporter and all that Prometheus knows at this point is service discovery data. Thus, with relabel_configs you can only manipulate data on the scale of hosts.

__name__ becomes available after scraping and there is still time before the metrics are processed and saved into the database. This is when metric_relabel_configs is happening, if you want to keep/drop by name - this is the place to do it:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: my_useless_metric.*
    action: drop
user_113445020
Related