Docker inside Docker in a Kubernetes Cluster

user_770205 0
Viewed 1243

I am trying to run a Jenkins pod in a Kubernetes cluster on bare metal.

I was trying to implement docker in docker as I need to build a Docker image in a pipeline so I mount /var/run/docker.sock as a volume into the container. Problem I faced was that I was receiving permission denied error each time a docker command run in the Jenkins pipeline.

I checked the /var/run/docker.sock ownership on the node 

srw-rw---- 1 root docker 0 Apr 10 19:47 /var/run/docker.sock

Then I checked it inside the running container:

srw-rw---- 1 root 116 0 Apr 10 21:33 /var/run/docker.sock

Now I am a bit confused. Why I see an ID? I checked the groupID of the docker group on the host. It is exactly 116. I guess logical assumption would be that ownership inside the container is exactly the same as on the host. Though the docker group on the host is not seeing as a docker group inside the container. Is that the correct assumption? Then tThe question is: what did I do wrong?

My Jenkins Dockerfile looks as following

FROM jenkins/jenkins:2.230-jdk11
....
UN apt-get update -qq  && apt-get install -y docker-ce docker-ce-cli containerd.io

#The following 3 commands I tried to fix the issue, but it did not help
RUN touch /var/run/docker.sock
RUN chown jenkins:docker /var/run/docker.sock
RUN usermod -aG docker jenkins

USER jenkins

I assume that the groups could be either - somehow mapped between host and guest OSs (not sure it works this way(does it?)) or - jenkins user should be added to the docker group on the host OS, but I failed to find how. Probably it could be done as suggested here with the command on a pod start, but then I would have to run the pod as a user that has rights to do so, which might be not the best idea. or - something else

Please advise, what is the advised way to have this working.

Thank you

PS The issue can be clearly fixed by having 

  securityContext:
    ..
    runAsGroup: 116
    ..

in the deployment definition, but it is not a valid solution.

chown kubernetes docker
1 Answers

There are a number of ways to run docker in kubernetes.

  1. Do you really need docker or do you just need to build a container? If you just want to build docker containers img, kaniko, and buildah will build docker images.
  2. Jenkins can create a kubernetes pod with multiple containers. Different stages can use different containers, and share the same build dir. See https://plugins.jenkins.io/kubernetes/
  3. Do you have a system you can use as a docker build host or jenkins agent? If only a few builds need docker then maybe it's worth it to use a normal agent or a docker build host.
  4. Actually use Docker in Docker. Danger Danger https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
  5. Use something like sysbox or kubevirt to run a VM.
user_128013990
Related