Jenkins on Kubernetes - Permission Denied

Viewed 3770

I'm trying to isntall Jenkins on Kubernetes using Helm 3 and following hte official instructions but running up against a permission issue.

---

apiVersion: v1
kind: Namespace
metadata:
  name: jenkins
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
  namespace: jenkins
spec:
  storageClassName: jenkins-pv
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 5Gi
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/jenkins-volume/

Then pull down the values.yaml file: wget https://raw.githubusercontent.com/jenkinsci/helm-charts/main/charts/jenkins/values.yaml

I adjust the adminPassword (this is a demo system): adminPassword: "mySecret"

Finally I change storageClass: to be storageClass: jenkins-pv

Output / Debug Logs

$ kubectl logs -n jenkins jenkins-0 init
disable Setup Wizard
/var/jenkins_config/apply_config.sh: 4: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/jenkins.install.UpgradeWizard.state: Permission denied


$ kubectl describe pod -n jenkins jenkins-0

Name:         jenkins-0
Namespace:    jenkins
Priority:     0
Node:         ip-172-31-40-127/172.31.40.127
Start Time:   Mon, 30 Nov 2020 10:37:19 +0000
Labels:       app.kubernetes.io/component=jenkins-controller
              app.kubernetes.io/instance=jenkins
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=jenkins
              controller-revision-hash=jenkins-57958b7d49
              statefulset.kubernetes.io/pod-name=jenkins-0
Annotations:  checksum/config: 2a4c2b3ea5dea271cb7c0b8e8582b682814d39f8e933e0348725b0b9a7dbf258
Status:       Pending
IP:           10.42.0.44
IPs:
  IP:           10.42.0.44
Controlled By:  StatefulSet/jenkins
Init Containers:
  init:
    Container ID:  containerd://64862ebd6791966db07981196d5dbd4c3b583d9e3e6543a31b252d19c2f9405b
    Image:         jenkins/jenkins:lts
    Image ID:      docker.io/jenkins/jenkins@sha256:980d55fd29a287d2d085c08c2bb6c629395ab2e3dd7547641035b4f126acc322
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      /var/jenkins_config/apply_config.sh
    State:          Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 30 Nov 2020 10:53:41 +0000
      Finished:     Mon, 30 Nov 2020 10:53:41 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 30 Nov 2020 10:48:29 +0000
      Finished:     Mon, 30 Nov 2020 10:48:29 +0000
    Ready:          False
    Restart Count:  8
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:        50m
      memory:     256Mi
    Environment:  <none>
    Mounts:
      /usr/share/jenkins/ref/plugins from plugins (rw)
      /var/jenkins_config from jenkins-config (rw)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_plugins from plugin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-zjzdt (ro)
Containers:
  jenkins:
    Container ID:
    Image:         jenkins/jenkins:lts
    Image ID:
    Ports:         8080/TCP, 50000/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --httpPort=8080
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      50m
      memory:   256Mi
    Liveness:   http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=5
    Readiness:  http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=3
    Startup:    http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=12
    Environment:
      POD_NAME:                  jenkins-0 (v1:metadata.name)
      JAVA_OPTS:                 -Dcasc.reload.token=$(POD_NAME)
      JENKINS_OPTS:
      JENKINS_SLAVE_AGENT_PORT:  50000
      CASC_JENKINS_CONFIG:       /var/jenkins_home/casc_configs
    Mounts:
      /run/secrets/chart-admin-password from admin-secret (ro,path="jenkins-admin-password")
      /run/secrets/chart-admin-username from admin-secret (ro,path="jenkins-admin-user")
      /usr/share/jenkins/ref/plugins/ from plugin-dir (rw)
      /var/jenkins_config from jenkins-config (ro)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_home/casc_configs from sc-config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-zjzdt (ro)
  config-reload:
    Container ID:
    Image:          kiwigrid/k8s-sidecar:0.1.275
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAME:           jenkins-0 (v1:metadata.name)
      LABEL:              jenkins-jenkins-config
      FOLDER:             /var/jenkins_home/casc_configs
      NAMESPACE:          jenkins
      REQ_URL:            http://localhost:8080/reload-configuration-as-code/?casc-reload-token=$(POD_NAME)
      REQ_METHOD:         POST
      REQ_RETRY_CONNECT:  10
    Mounts:
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_home/casc_configs from sc-config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-zjzdt (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  plugins:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  jenkins-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      jenkins
    Optional:  false
  plugin-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  jenkins-home:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins
    ReadOnly:   false
  sc-config-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  admin-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  jenkins
    Optional:    false
  jenkins-token-zjzdt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  jenkins-token-zjzdt
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From                       Message
  ----     ------     ----                ----                       -------
  Normal   Scheduled  <unknown>           default-scheduler          Successfully assigned jenkins/jenkins-0 to ip-172-31-40-127
  Normal   Pulled     15m (x4 over 16m)   kubelet, ip-172-31-40-127  Successfully pulled image "jenkins/jenkins:lts"
  Normal   Created    15m (x4 over 16m)   kubelet, ip-172-31-40-127  Created container init
  Normal   Started    15m (x4 over 16m)   kubelet, ip-172-31-40-127  Started container init
  Normal   Pulling    14m (x5 over 16m)   kubelet, ip-172-31-40-127  Pulling image "jenkins/jenkins:lts"
  Warning  BackOff    74s (x71 over 16m)  kubelet, ip-172-31-40-127  Back-off restarting failed container

3 Answers

I see that this happens when using hostPath Minikube one node cluster, like in the documentation. The issue is because the /data/jenkins-volume folder in the Minikube node is created with root ownership.

So, if you don't want to run as root with the runAsUser: 0 you can just change the permissions in the /data/jenkins-volume entering into the node with:

$ minikube ssh

$ sudo chown -R 1000:1000 /data/jenkins-volume

Once you do that you can create the pv and deploy Jenkins with Helm with values:

runAsUser: 1000
fsGroup: 1000 

It worked for me.

I had the same issue. As Mariusz suggested in his comment:

Does it work when you change runAsUser in values.yaml to 0? If not could you provide more info about your setup?

I changed to runAsUser: 0 and this helped the jenkins pod to start properly.

Change containerSecurityContext to:

containerSecurityContext: runAsUser: 0 runAsGroup: 0 readOnlyRootFilesystem: false allowPrivilegeEscalation: true

Related