Relabeling in Prometheus

user_695761 0
Viewed 12871

Setup

Prometheus node exporter is registered as a service with consul agent with various tags. Example service definition provided to consul agent:

{
  "service":{
      "id": "server-stats",
      "name": "server-stats",
      "tags": [
        "a=1_meow",
        "b=2_woof",
        "c=3_moo",
        "monkey"
      ],
      "port": 9100,
      "checks": [
        {
          "name": "Process #1",
          "script": "/path/to/healthcheck/script.sh",
          "interval": "5s"
        }
      ]
    }
}

Prometheus is set to look for this server-stats service and use the configuration (host address and port) provided by Consul to scrape stats from servers. The above tags are available as a comma separated list in __meta_consul_tags that can be used for relabeling.

Prometheus relabeling configuration:

relabel_configs:
- source_labels: [__meta_consul_tags]
  separator:     ','
  #regex:         '(.+)=(.+)'
  regex:         '([a-z_]+)=([a-z_]+|\d+)'
  target_label:  ${1}
  replacement:   ${2}

Issue

I am trying to expose tags to Prometheus so that we can get stats and graphs based on labels. Keeping the above service configuration in mind, I would like each metric to have following labels in addition to whatever Prometheus does internally: a=1_meow, b=2_woof, c=3_moo and ignore monkey because it is just a string. I can remove monkey from my list of tags if there is a solution that requires = to be there. The relabel configuration written above is not leading to exposing any tag at all and seems to be getting ignored. Running Prometheus with log level set to debug is also not yielding anything.

Relevant docs

consul prometheus monitoring
3 Answers

The following relabeling rule can be used for extracting a particular tag from __meta_consul_tags and placing it to destination_label:

relabel_configs:
- source_labels: [__meta_consul_tags]
  regex:         '.*,tag=([^,]+),.*'
  target_label:  destination_label

There is no need to specify the replacement option because it defaults to $1.

If multiple tags must be extracted from __meta_consul_tags into multiple labels, then just repeat the relabeling rule per each needed tag. For example, the following relabeling rules extract a tag to a label and b tag to b label:

relabel_configs:
- source_labels: [__meta_consul_tags]
  regex: '.*,a=([^,]+),.*'
  target_label: a
- source_labels: [__meta_consul_tags]
  regex: '.*,b=([^,]+),.*'
  target_label: b

More information about relabeling rules in Prometheus is available in this article.

user_2749370
Related