Permission denied on /etc/prometheus/prometheus.yml; cannot deploy prom/prometheus

user_2927841 0
Viewed 5469

I'm using an NFS mount to provide the /etc/prometheus/prometheus.yml (default) config file to prom/prometheus docker image all via Ansible. When the container is deployed, I'm getting the following error in the container logs, and the container restarts after a few seconds.

level=error ts=2020-10-28T16:01:04.432Z caller=main.go:290 msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" err="open /etc/prometheus/prometheus.yml: permission denied"

I can browse the mounted filesystem on my docker host (a Raspberry Pi 4), touch files and read prometheus.yml as the user that launches the container.

Below are the relevant tasks from my playbook, and the issue is the same when deploying the container from the CLI without the playbook with the remote filesystem mounted to at /mnt/prometheus, and passed to the container as a volume at /etc/prometheus

docker run -p 9090:9090 -v /mnt/prometheus:/etc/prometheus prom/prometheus

prometheus/tasks/main.yml (become: yes is set in the playbook that calls this role)

  - name: "Create mountpoint"
    file: 
        path: "{{ prometheus_mount_path }}"
        state: directory
        mode: 0777
        owner: root
        group: users
        

  - name: "Mount nfs drive for prometheus filesystem"
    mount: 
        path: "{{ prometheus_mount_path }}"
        src: "{{ nfs_server }}:{{ prometheus_nfs_path }}"
        state: mounted
        fstype: nfs
        
  - name: "Create prometheus.yml in mountpoint from template"
    template: 
        src: prometheus.yml.j2
        dest: "{{ prometheus_mount_path }}/prometheus.yml"

        
  - name: "Deploy prometheus container"
    docker_container:
        name: prometheus
        image: prom/prometheus:latest
        restart_policy: always
        state: started
        network_mode: host
        hostname: prometheus
#        exposed_ports: 9090
        published_ports: 9090:9090
        user: 995:1002
        mounts: 
        volumes:
            - "{{ prometheus_mount_path }}:/etc/prometheus"
        comparisons:
            '*': ignore
            env: strict

Any idea what would cause or how to resolve the permission denied issue from the container?

Update: I tested by sharing a directory on the docker host with the container. This was successfully shared. Points to an NFS issue, but I'm struggling to figure that out.

prometheus ansible docker nfs
2 Answers

I had a similar issue, although not with NFS but with mounting a simple directory. For me the answer in this question fixed it: docker-compose: directory permission errors on bind mount.

Apparently Prometheus Docker is using the user nobody, setting folder permissions to nogroup worked for me:

chgrp -R nogroup /mnt/prometheus

So in your Ansible playbook:

- name: fix permissions for prometheus mnt folder
  file:
    path: '/mnt/prometheus'
    group: nogroup
    recurse: yes
  become: yes
user_85131830

In my particular case, I found the issue was a function of NFS squashing. My NFS exporter - an older Synology NAS - didn't allow me to disable squashing entirely. If it had, @nehtor.t's answer probably would have helped. Instead I had to use the "map all users to admin" setting in the NFS GUI, which allowed connections the appropriate permissions on the NFS share to appropriately chown.

user_29278410
Related