filter/exclude prometheus metric based on particular label combination in open-telemetry

user_1958107 0
Viewed 67

I have a use case, where I wanted to exclude (filter) metric with particular label combination, I am using otel-collector for collection, processing and exporting metrics

however when I try to write a config it remove/exculde/fiter that metric completely with other label combination set.

.i.e.

so according to otel Config the receivers are configured as

receivers:
   prometheus:
    config:
      scrape_configs:
        - job_name: "otel-collector"
          scrape_interval: 30s
          static_configs:
            - targets: ["localhost:9001"]

which gets metric in question as

# TYPE node_supervisord_up gauge
node_supervisord_up{group="serviceA",name="serviceA"} 1
node_supervisord_up{group="serviceB",name="serviceB"} 1
node_supervisord_up{group="serviceC",name="serviceC"} 1
node_supervisord_up{group="ServiceD",name="serviceD"} 0
node_supervisord_up{group="ServiceE",name="serviceE"} 1

Now my intention is to remove node_supervisord_up{group="ServiceD",name="serviceD"} 0 before sending to central Metric aggregator

so I tried, below config of otel-collector

processors:
  exclude:
        match_type: expr
        expressions:
        - MetricName == "node_supervisord_up" && Label("name") == "serviceD"

Refered here https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md

however this results in complete exclude of metric name node_supervisord_up for other set of services label combination.

what configuration could help here to just exculde node_supervisord_up{group="ServiceD",name="serviceD"} 0 before sending to central aggregator from the client.

open-telemetry
1 Answers

Try to drop it using metric_relabel_configs. Metric relabeling is applied to samples as the last step before ingestion.

receivers:
   prometheus:
    config:
      scrape_configs:
        - job_name: "otel-collector"
          scrape_interval: 30s
          static_configs:
            - targets: ["localhost:9001"]
          metric_relabel_configs:
            - source_labels: [ group ]
              regex: '^ServiceD$'
              action: drop
user_108293000
Related