We have a fleet of pods in AKS running a CI/CD system (GitHub Actions runners) and each of them has a 32 GiB Azure Disk (Standard SSD) attached with a PVC, where we mount /var/lib/docker, so that the layers of GitHub actions based on docker images are cached (we use DinD and we start the docker daemon in a script when the pods start). The pods are running as a StatefulSet and the PVs attached using volumeClaimTemplates.
About, roughly, 30% of the time that a pod is started, trying to access /var/lib/docker results in value too large for the defined data type. As consequence, the docker daemon fails to start. Below is the log from the docker daemon
When trying chmod to change permissions for /var/lib/docker from the command line, the error is the same.
Manually killing the pod and recreating it usually solves the problem, as it is an intermittent problem and does not appear to be following any pattern. Looking into the cluster logs shows that all PVs are being attached with no issues and no errors are being shown.
The image we are using for the runners is based on ubuntu:focal and we are installing the following packages:
gnupg2 gnupg apt-transport-https ca-certificates software-properties-common pwgen git make curl wget zip libicu-dev build-essential libssl-dev git rsync lsb-release jq nodejs npm openjdk-16-jdk
OS information:
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
Platform: x86_64
Kubernetes version: 1.20.9
Research suggests a 32-bit application attempting to access either a large file or a very large number of files but I would expect such issue would occur always and not intermittently, as it is the case.
Any hints as to what could be behind this problem?
UPDATE As additional information, we are using sysbox (CE, version 0.4.1) to run docker in docker. We discovered that sysbox is performing shiftfs in /var/lib/docker, which should not be happening, so potentially a problem with sysbox.
