docker cadvisor container failing with machine_libipmctl.go:62] There are no NVM devices

user_14096639 0
Viewed 417

I'm trying to implement Prometheus for docker swarm by creating docker cadvisor as a service, but the docker logs shows 1 machine_libipmctl.go:62] There are no NVM devices! I couldn't see any metrics.

my docker-compose file

version: '3'

services:
  cadvisor:
    image: google/cadvisor:v0.39.3
    ports:
      - '2200:2200'
    command:
      - "--housekeeping_interval=30s"
      - "--docker_only=true"
      - "--disable_metrics=percpu,sched,tcp,udp,disk,diskIO,accelerator,hugetlb,referenced_memory,cpu_topology,resctrl"
    volumes:
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
      - /sys:/sys:ro
      - /var/run:/var/run:rw
      - /:/rootfs:ro
      - /sys/fs/cgroup:/cgroup:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /etc/localtime:/etc/localtime:ro

    devices:
      - /dev/kmsg:/dev/kmsg

Here's what i see inside the container


/ # ps -ef
PID   USER     TIME  COMMAND
    1 root      3:40 /usr/bin/cadvisor -logtostderr --housekeeping_interval=30s --docker_only=true --disable_metrics=percpu,sched,tcp,udp,disk,diskIO,accelerator,hugetlb,referenced_memo
18108 root      0:00 /bin/sh

/ # wget http://localhost:8080/healthz
Connecting to localhost:8080 (127.0.0.1:8080)
saving to 'healthz'
healthz              100% |******************************************************************************************************************************************|     2  0:00:00 ETA
'healthz' saved
/ # wget http://localhost:8080/metrics
Connecting to localhost:8080 (127.0.0.1:8080)
saving to 'metrics'
metrics              100% |******************************************************************************************************************************************| 9405k  0:00:00 ETA
'metrics' saved
18114 root      0:00 ps -ef

i port forwarded cadvisor port 8080 to 2200

curl -Ik http//:localhost:2200/metrics
curl: (6) Could not resolve host: http; Name or service not known

What I'm doing wrong here? I can see a service running with port 2200 and prometheus.yml config file

    scheme: http
    metrics_path: /metrics
    static_configs:
      - targets: ['server_name:2200']
 dial tcp X.X.X.XX:2200: connect: connection refused
cadvisor docker-compose docker
1 Answers
version: '3'

services:
  cadvisor:
    image: google/cadvisor:v0.39.3
    ports:
      - '2200:2200'
    command:
      - "--housekeeping_interval=30s"
      - "--docker_only=true"
      - "--disable_metrics=percpu,sched,tcp,udp,disk,diskIO,accelerator,hugetlb,referenced_memory,cpu_topology,resctrl"
    volumes:
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
      - /sys:/sys:ro
      - /var/run:/var/run:rw
      - /:/rootfs:ro
      - /sys/fs/cgroup:/cgroup:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /etc/localtime:/etc/localtime:ro

    devices:
      - /dev/kmsg:/dev/kmsg
    privileged: true

privileged: true fixed the issue, you can setup cadvisor in all docker swarm cluster nodes by docker create service

docker service create --name cadvisor -l prometheus-job=cadvisor \
    --mode=global --publish published=8800,target=8080 \
    --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock,ro \
    --mount type=bind,src=/,dst=/rootfs,ro \
    --mount type=bind,src=/var/run,dst=/var/run \
    --mount type=bind,src=/sys,dst=/sys,ro \
    --mount type=bind,src=/var/lib/docker,dst=/var/lib/docker,ro \
     google/cadvisor:v0.39.3 -docker_only
user_140966390
Related