I'm having issues with kubelet removing docker images because it believes the disk is full:
Dec 29 18:00:14 hostname kubelet: I1229 18:00:14.548513 13836 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 85% which is over the high threshold (85%). Trying to free 2160300032 bytes down to the low threshold (80%).
However, the partition that docker uses is 1TB and has plenty of space:
$ docker info
...
Docker Root Dir: /scratch/docker
$ df -k /scratch
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 976283900 234476968 741806932 25% /scratch
It seems kubelet is finding the disk usage for my main partition on /, which also happens to be the partition kubelet itself is installed on:
$ df -k /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nvme0n1p2 52403200 44032936 8370264 85% /
So where does kubelet get information about available disk space? I assumed it was using the docker daemon, but based on what I'm seeing the two apps are looking at different partitions. Is there a configuration I can set, or is does it just default to its own partition when doing the disk space check?
This is using Kubernetes 1.17.4 on RedHat 7 and docker 18.06.