container's runAsUser breaks non-root policy in prometheus helm chart (wireguard sidecar)

user_10065674 0
Viewed 453

I want to add a wireguard sidecar container to prometheus stack installed via helm chart such that I can fetch client connected via vpn. I'm using the helm chart from prometheus-community/kube-prometheus-stack with modified values.yml. To integrate wireguard I added a wireguard container in the values.yml like this:

...
containers:
    - name: "wireguard"
      image: "lscr.io/linuxserver/wireguard:latest"
      volumeMounts:
        - name: wireguard-config
          mountPath: /config
          readOnly: true
        - name: wireguard-run
          mountPath: /run
      securityContext:
        runAsGroup: 0
        runAsUser: 0
        privileged: true
        capabilities:
          add:
            - NET_ADMIN
            - SYS_MODULE

but when I start the containers, I get the following error:

Normal   Pulled     4s               kubelet            Successfully pulled image "lscr.io/linuxserver/wireguard:latest" in 500.578587ms
Warning  Failed     3s (x3 over 4s)  kubelet            Error: container's runAsUser breaks non-root policy (pod: "XX", container: wireguard)
Normal   Pulled     3s               kubelet            Successfully pulled image "lscr.io/linuxserver/wireguard:latest" in 456.879479ms

As wireguard needs to be able to change network interfaces, it needs the root privileges. If I don't run the container with root privileges I get the following:

...
SOME OTHER PERMISSION ERROS
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise coredns: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise wireguard: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise coredns: warning: unable to spawn ./run - waiting 10 seconds

What I tried is modifying the podSecurityPolicy to allow running containers as root in the prometheus values.yml. I was hoping that I could simply run the container as root (for testing at least):

podSecurityPolicy:
    allowedCapabilities: 
       - runAsUser: RunAsAny
       - NET_ADMIN
       - SYS_MODULE
    allowedHostPaths: []
    volumes: []

This didn't change anything (am I doing this correctly?)

How would I allow running a sidecar container as root? Or is there a way to run wireguard without root priv?

wireguard helm3 prometheus kubernetes-helm kubernetes
1 Answers

  1. there is no psp in the newer kube-prometheus-stack. see https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md - search for PodSecurityPolicies - "From 27.x to 28.x This version disables PodSecurityPolicies by default because they are deprecated in Kubernetes 1.21 and will be removed in Kubernetes 1.25."

  2. we need to play with securityContext. it seems that you cannot have a pod running as non-root with a container (or init container) running as root. this is the error "kubelet Error: container's runAsUser breaks non-root policy ".

so we need to specify for the whole pod the securityContext run as root. for this, you may use a values.yaml for helm chart like below. the effect is the pod prometheus-prometheus-stack-kube-prom-prometheus-0 will run all its containers as root. not ideal but it works.

prometheus:
  prometheusSpec:

    securityContext:
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
      fsGroup: 0

#    initContainers:
#      - name: "chmod"
#        image: alpine:3.16.0
#        command:
#        - "/bin/sh"
#        - "-c"
#        - "chmod 777 /prometheus"
#        volumeMounts:
#        - name: prometheus-prometheus-stack-kube-prom-prometheus-db
#          mountPath: /prometheus

[...]
user_195288100
Related