I have a host with the ip 10.0.0.6 on a private network that runs prometheus.
This and several other hosts in the network run a node-exporter via docker-compose where {{ private_ip }} is automatically replaced by the actual private ip:
node-exporter:
image: prom/node-exporter:v1.3.1
pid: host
volumes:
- '/:/host:ro,rslave'
command:
- '--path.rootfs=/host'
expose:
- 9100
ports:
- {{ private_ip }}:9100:9100
restart: unless-stopped
This is my prometheus.yml:
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: 'node-exporters'
scheme: http
basic_auth:
username: {{ node_exporter_auth_user }}
password: {{ node_exporter_auth_password }}
hetzner_sd_configs:
- role: "hcloud"
bearer_token: "{{ hcloud_prometheus_token }}"
relabel_configs:
- source_labels: [__meta_hetzner_hcloud_label_node_exporter]
action: keep
regex: true
- source_labels: [__meta_hetzner_hcloud_label_node_exporter, __meta_hetzner_hcloud_private_ipv4_mynet]
action: replace
regex: true;(.*)
target_label: __address__
replacement: '$1:9100'
- action: labelmap
regex: __meta_hetzner_hcloud_(.+)
- action: labelmap
regex: __meta_hetzner_server_(.+)
Prometheus is able to scrape all other hosts but not its own host. The target https://10.0.0.6:9100/metrics is correctly discovered and shows up in prometheus' web ui under the targets.
However, its status keeps UNKNOWN because it is never successfully scraped.
What I tried
On 10.0.0.6 I tried to put node-exporter on the host network:
node-exporter:
network_mode: host
For the prometheus service I tried to add
prometheus:
extra_hosts:
- "host.docker.internal:host-gateway"
I also ensured that I can curl the metrics successfully (but prometheus cannot) :
local$ ssh 10.0.0.6
remote$ curl -u someuser:correctpassword http://10.0.0.6:9100/metrics
# shows the metrics