docker is full, all inodes are used

user_8030268 0
Viewed 14558

got huge problem, all my inodes seems to be used. I've cleaned all volumes unused Cleaned all container and images with command -> docker prune

but still it seems that it stay full :

Filesystem      Inodes   IUsed  IFree IUse% Mounted on
none           3200000 3198742   1258  100% /
tmpfs           873942      16 873926    1% /dev
tmpfs           873942      13 873929    1% /sys/fs/cgroup
/dev/sda1      3200000 3198742   1258  100% /images
shm             873942       1 873941    1% /dev/shm
tmpfs           873942       1 873941    1% /sys/firmware

docker info

Containers: 5
 Running: 3
 Paused: 0
 Stopped: 2
Images: 23
Server Version: 17.06.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 53
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.668GiB
Name: serveur-1
ID: CW7J:FJAH:S4GR:4CGD:ZRWI:EDBY:AYBX:H2SD:TWZO:STZU:GSCX:TRIC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

The only thing i think can do this, is a build i'm doing on this machine. This build runs a npm install with many files. Can these files stays on server ? is there any chance i have to delete these temporary files ?

docker
5 Answers

I had the same problem. Had Jenkins running inside Docker with a volume attached to it. After a few weeks Jenkins told me "npm WARN tar ENOSPC: no space left on device". After some googling I found out that all inodes are taken with sudo df -ih. With sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n I could locate the folder using up all the inodes and it was a certain build with npm. Deleted that folder and now I'm good to go again.

user_106704140

In my case it was dangling build cache because removing dangling images does not solve the issue.

This cache can be removed by following command: docker system prune --all --force, but be careful maybe you still need some volumes or images.

user_15920080

This can also be an effect of a lot of stopped containers, for example if there's a cron job running that use a container, and the docker run commandline used does not include --rm - in that case, every cron invocation will leave a stopped container on the filesystem.

In this case, the output of docker info will show a high number under Server -> Containers -> Stopped.

To cure this:

  1. docker container prune
  2. Add --rm to your docker run command line in the cron job.
Erik_Forsberg_86750
Related