prometheus cannot scrape node-exporter on own host

Viewed 324

I have a host with the ip 10.0.0.6 on a private network that runs prometheus.

This and several other hosts in the network run a node-exporter via docker-compose where {{ private_ip }} is automatically replaced by the actual private ip:

  node-exporter:
    image: prom/node-exporter:v1.3.1
    pid: host
    volumes:
      - '/:/host:ro,rslave'
    command:
      - '--path.rootfs=/host'
    expose:
      - 9100
    ports:
      - {{ private_ip }}:9100:9100
    restart: unless-stopped

This is my prometheus.yml:

global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'node-exporters'
    scheme: http
    basic_auth:
      username: {{ node_exporter_auth_user }}
      password: {{ node_exporter_auth_password }}
    hetzner_sd_configs:
      - role: "hcloud"
        bearer_token: "{{ hcloud_prometheus_token }}"
    relabel_configs:
      - source_labels: [__meta_hetzner_hcloud_label_node_exporter]
        action: keep
        regex: true
      - source_labels: [__meta_hetzner_hcloud_label_node_exporter, __meta_hetzner_hcloud_private_ipv4_mynet]
        action: replace
        regex: true;(.*)
        target_label: __address__
        replacement: '$1:9100'
      - action: labelmap
        regex: __meta_hetzner_hcloud_(.+)
      - action: labelmap
        regex: __meta_hetzner_server_(.+)

Prometheus is able to scrape all other hosts but not its own host. The target https://10.0.0.6:9100/metrics is correctly discovered and shows up in prometheus' web ui under the targets.

However, its status keeps UNKNOWN because it is never successfully scraped.

What I tried

On 10.0.0.6 I tried to put node-exporter on the host network:

  node-exporter:
    network_mode: host

For the prometheus service I tried to add

  prometheus:
    extra_hosts:
      - "host.docker.internal:host-gateway"

I also ensured that I can curl the metrics successfully (but prometheus cannot) :

local$ ssh 10.0.0.6
remote$ curl -u someuser:correctpassword http://10.0.0.6:9100/metrics
# shows the metrics
0 Answers
Related