How to relabel Kubernetes metrics with dynamic pod URLs in Prometheus?

Viewed 461

I am trying to discover both pods and nodes on the same job using kubernetes_sd_configs and use their labels together. I have blackbox-exporter on multiple pods in my cluster on different nodes, my goal is to monitor each of them, but I am having trouble with the metric relabelling. I am trying to achieve something like this:

http://<pod-ip1>:8082/metrics?instance=<node-ip1>
http://<pod-ip2>:8082/metrics?instance=<node-ip1>
http://<pod-ipN>:8082/metrics?instance=<node-ip1>
.
.
http://<pod-ip1>:8082/metrics?instance=<node-ip2>
http://<pod-ip2>:8082/metrics?instance=<node-ip2>
http://<pod-ipN>:8082/metrics?instance=<node-ip2>
.
.

My current configuration looks like the following, but the pod URL is missing:

- job_name: 'kubernetes-pods'
        metrics_path: /probe
        params:
          module: [ping]

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: <pod_should_be_here>:9115
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
0 Answers
Related