docker-compose --build taking forever, takes massive amount of space

user_7211014 0
Viewed 1873

Problem(2):

Performing a docker-compose --build or docker-compose up --build results in all containers taking very long to build. Also docker appears to be creating a very large tmp file in /tmp, which I believe is the current directory written to this file.

Trouble shooting:

  • tried upgrading docker to 20.10.6
  • Doing a watch df -H shows the main partition fill up, and then the system does nothing (docker-compose --build process hung, nothing happens).
  • I performed a docker system prune -a, that removed about 50GB.
  • The host has 200GB partition set up for / , and this is where all the docker volumes live. I extended this partition to 500GB, and tried to build again, at this point the first container image FINNALLY builds (took 20min). After that the next image starts, which takes 20min to build as well, etc for each other container, took about 2h to build everything.
  • I found the source of space exhaustion. Inside /tmp there is a file called "tmpsx30lncc" (i assume random chars at the end), its size is growing and matches the size consumed during the build process. Running file tmpsx30lncc shows its a POSIX tar archive. I have read online that this file is an archive of my current directory where docker-compose build was run. Transfer rate appears to vary. Based on watching the file increase, it is anywhere form 50-250MB a second.
  • At some point (I believe when all of the contents are archived to the tmp archive file in /tmp, df -H shows the size being returned slowly, while each container gets to it's build step.
  • I did a du --block-size=M -a / | sort -n -r | head -n 20, the main culprits for large files the /tmp file (full size of directory where docker-compose was run), my docker-registry folder (~70GB), and lastly gitea (4GB), other containers have other folders that add up as well. The main directory where docker-compose is copying all the files is ~76GB.

System

  • OS: Ubuntu 18.04
  • Docker version: 19.03.3
  • Containers trying to build in docker-compose: gitea, postgres, drone, drone-runner, docker-registry
  • volumes: Are all located in my / partition in various locations. Gitea will build at some point, but the next image will exhibit the same behavior and take just as long and fill up the hard drive.

docker-info:

    Client:
     Context:    default
     Debug Mode: false
     Plugins:
      app: Docker App (Docker Inc., v0.9.1-beta3)
      buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
      scan: Docker Scan (Docker Inc., v0.7.0)
    
    Server:
     Containers: 0
      Running: 0
      Paused: 0
      Stopped: 0
     Images: 0
     Server Version: 19.03.3
     Storage Driver: overlay2
      Backing Filesystem: extfs
      Supports d_type: true
      Native Overlay Diff: true
     Logging Driver: json-file
     Cgroup Driver: cgroupfs
     Plugins:
      Volume: local
      Network: bridge host ipvlan macvlan null overlay
      Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
     Swarm: inactive
     Runtimes: runc
     Default Runtime: runc
     Init Binary: docker-init
     containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
     runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
     init version: fec3683
     Security Options:
      apparmor
      seccomp
       Profile: default
     Kernel Version: 4.15.0-141-generic
     Operating System: Ubuntu 18.04.5 LTS
     OSType: linux
     Architecture: x86_64
     CPUs: 10
     Total Memory: 15.64GiB
     Name: taxmd01-gitjenkins-v
     ID: ZZYO:KANZ:3AH3:UO4U:HFHC:G6OW:4UFQ:S7Q5:QNVC:O5QH:33P7:MRMN
     Docker Root Dir: /var/lib/docker
     Debug Mode: false
     Registry: https://index.docker.io/v1/
     Labels:
     Experimental: false
     Insecure Registries:
      127.0.0.0/8
     Live Restore Enabled: false

How to reproduce

  • Use same system and versioning as above
  • Here is my docker-compose.yaml (I have adjusted IPs, domains, passwords, and secrets), to get this to work you will need to adjust IP's ports, certs, and directory structure.
  • Configure all IP addresses on one interface with netplan in ubuntu
  • To deploy recreate directory structure, and run docker-compose build in the same directory as the docker-compose.yaml file below. I have concerns that if you actually set everything up correctly, you probably wont run into the same issues I am as my images / volumes are quite large due to use over time.
    version: "3.7"
    
    services:
      gitea:
        build:
          context: ./
          dockerfile: services/gitea/gitea.Dockerfile
        volumes:
          - ./services/gitea/data:/data
          - /etc/timezone:/etc/timezone:ro
          - /etc/localtime:/etc/localtime:ro
        ports:
          - 10.10.10.31:22:22
          - 10.10.10.31:80:3080
          - 10.10.10.31:443:3443
        environment:
          - USER_UID=1000
          - USER_GID=1000
          - DB_TYPE=postgres
          - DB_HOST=postgres:5432
          - DB_NAME=myDBname
          - DB_USER=myUser
          - DB_PASSWD=myPassword
      postgres:
        image: postgres:latest
        volumes:
          - ./services/postgres/data:/var/lib/postgresql/data
        environment:
          - POSTGRES_USER=myUser
          - POSTGRES_PASSWORD=myPassword
          - POSTGRES_DB=myDBname
      drone:
        build:
          context: ./
          dockerfile: services/drone/drone.Dockerfile
        ports:
          - 10.10.10.28:80:80
          - 10.10.10.28:443:443
        volumes:
          - ./services/drone/data:/data
        environment:
          DRONE_GITEA_SERVER: "https://gitea2.example.internal"
          DRONE_GITEA_CLIENT_ID: 023f302d-ffff-22b3-83fc-898d9b68fbe1
          DRONE_GITEA_CLIENT_SECRET: MySecretab897a797c979d79779e79f
          DRONE_USER_CREATE: username:drone-admin,machine:false,admin:true,token:MyToken3892183902183902183012
          DRONE_SERVER_PROTO: https
          DRONE_SERVER_HOST: "drone2.example.internal"
          DRONE_RPC_SECRET: MyRPCSecretab897a797c979d79779e79f
          DRONE_TLS_CERT: /cert/drone.crt
          DRONE_TLS_KEY: /cert/drone.key
          DRONE_LOGS_DEBUG: "false"
          DRONE_LOGS_TRACE: "false"
      drone-runner:
        build:
          context: ./
          dockerfile: services/drone-runner/drone-runner.Dockerfile
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
        environment:
          DRONE_RPC_PROTO: https
          DRONE_RPC_HOST: drone
          DRONE_RPC_SECRET: MyRPCSecretab897a797c979d79779e79f
          DRONE_RUNNER_CAPACITY: 4
          DRONE_RUNNER_NAME: drone-runner
          DRONE_RPC_DUMP_HTTP: "false"
          DRONE_RPC_DUMP_HTTP_BODY: "false"
      docker-registry:
        build:
          context: ./
          dockerfile: services/docker-registry/docker-registry.Dockerfile
        ports:
          - 10.10.10.27:443:443
        volumes:
          - ./services/docker-registry/data:/var/lib/registry
        environment:
          REGISTRY_HTTP_ADDR: 0.0.0.0:443
          REGISTRY_HTTP_TLS_CERTIFICATE: /cert/docker-registry.crt
          REGISTRY_HTTP_TLS_KEY: /cert/docker-registry.key

Here are are each of the docker files:

srv/ci/services# cat gitea/gitea.Dockerfile
from gitea/gitea:latest
copy ./ca/* /usr/local/share/ca-certificates/
run update-ca-certificates
copy ./services/gitea/cert /cert
run chmod 700 /cert && chown -R myUser:myUser /cert

/srv/ci/services# cat drone/drone.Dockerfile
from drone/drone:1
run apk update && apk add --no-cache ca-certificates
copy ./ca/ca.crt /usr/local/share/ca-certificates/
run update-ca-certificates
copy ./services/drone/cert /cert
run chmod 700 /cert

/srv/ci/services# cat drone-runner/drone-runner.Dockerfile
from drone/drone-runner-docker:1
run apk update && apk add --no-cache ca-certificates
copy ./ca/ca.crt /usr/local/share/ca-certificates/
run update-ca-certificates

/srv/ci/services# cat docker-registry/docker-registry.Dockerfile
from registry
run apk update && apk add --no-cache ca-certificates
copy ./ca/ca.crt /usr/local/share/ca-certificates/
run update-ca-certificates
copy ./services/docker-registry/cert /cert/
run chmod 700 /cert

Here is the directory structure

enter image description here

So forgive me, I am not an expert on docker, and I did not build this system initially, its used for our dev team. This system has been around for 2 years now, and has grown a lot space utilization wise. I can only assume the problem is the databases of the containers are getting filled up, or the docker-registry has many images that are taking up too much space.

Questions:

  • Why are each of my containers taking 20min to build each? (I assume because it needs to copy over all of the files located in docker-compose file is.)
  • Why is there this massive tmp file in my /tmp dir, and why does my system transfer to this file so slowly? Is there anyway to stop docker from creating this file?
  • Any advice on how I can reduce these ridiculously long build times?
docker-registry docker-compose docker build
2 Answers

Docker uses a client/server model. You don't actually directly launch containers or build images; instead, you make an HTTP connection to the Docker daemon (usually over a Unix socket) and give it instructions on what to do.

More specifically, when you go to build an image, the Docker daemon doesn't directly read the filesystem. Instead, docker build and docker-compose build both make a tar file of the context directory and send it as the body of the HTTP POST request to start the build. When the Docker daemon receives the file, it needs to buffer it on disk and then unpack it to actually start the build.

You mention that, in a subdirectory of the build: {context: } directory, you have a couple of large data directories (70 GB for one application, 4 GB for another). Even if this content isn't actually used in the build, it still goes through this network-transfer process (even on a local system). This takes time to transfer across the socket (20 minutes to move 75 GB is believable) and it needs to be saved and unpacked (the very large file in /tmp you see).

You can create a .dockerignore file that names files that need to be omitted from the context. This needs to be in the context: directory, even if the dockerfile: is somewhere else. With the directory structure you show, you can omit all of the data directories:

# /srv/ci/Dockerfile
services/*/data

The reverse of this is that files you include in .dockerignore can't be COPYed into the image. This only affects the image build process and has no effect on volumes:; you could bind-mount the omitted directory into the container at runtime, even if it shouldn't be included in the image.

user_100081730

for me it was I didn't add .dockerignore and it copied whole . directory as a context for building

user_35743790
Related