Use label in regex for Prometheus relabel configs

Question

I'm trying to filter out some targets by network in my dockerswarm_sd_configs. It's working if I hardcode the network name in the config file but I want to set the filter from a docker label (in my docker compose).

This is my config file form Prometheus :

 # Create a job for Docker Swarm containers.
  - job_name: 'dockerswarm'
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: tasks
        port: 80

    relabel_configs:
      # Only keep containers that should be running.
      - source_labels: [__meta_dockerswarm_task_desired_state]
        regex: running
        action: keep
      # Only keep containers that have a `prometheus-job` label.
      - source_labels: [__meta_dockerswarm_service_label_prometheus_job]
        regex: .+
        action: keep
      # Use the prometheus-job Swarm label as Prometheus job label.
      - source_labels: [__meta_dockerswarm_service_label_prometheus_job]
        target_label: job
      # Keep port
      - source_labels: [__meta_dockerswarm_service_label_port]
        target_label: port
      # Keep network
      - source_labels: [__meta_dockerswarm_service_label_scrape_network]
        target_label: network
      # Relabel address to inject custom port value
      - source_labels: [__address__, __meta_dockerswarm_service_label_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      # Check network
      - source_labels: [__meta_dockerswarm_network_name]
        # Works with hardcoded value but not an other label
        regex: network
        action: keep

Is there a way to use a label as a variable in the regex field ?

user_747047 · Answer

It's not the cleanest answer but I ended up doing a separated config file for my two prometheus (one for each docker network). The last step now looks like this:

      # Check network
      - source_labels: [__meta_dockerswarm_network_name,__meta_dockerswarm_service_label_prometheus_scrape_network]
        regex: network1;network1
        action: keep

The first line (source_labels) gather the "meta network name" (the network name scraped from docker for this target) and the "label network", which is given in my docker stack:

      labels:
        # Prometheus Swarm
        - prometheus-job=myJobsName
        - prometheus-port=3000
        - prometheus-scrape-network=network1
        - prometheus-env=production

The second line checks that the label equals network1;network1 so it will filter out the "duplicate" target on the other network (because the other one will be equals to network2;network1).

Use label in regex for Prometheus relabel configs

1 Answers