I am using docker executor in Gitlab to build Ci/CD pipelines. And frequently I am facing this particular issue. It started happening all of a sudden and all the builds started breaking.
I did some googling and the only root cause I could find is people asking to restart Docker but that should not be the end solution. Getting error while running docker run :Error response from daemon
Error I am seeing -
Running with gitlab-runner 13.6.0 (8fa89735)
on docker-runner XBb4YMoN
Preparing the "docker" executor
00:12
Using Docker executor with image registry.gitlab.systems.co.uk/devopstooling/dockerimages:node-14 ...
ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "dkfhdfdsjkhdfjlk;jfl;ndbjkfbjkdsbfj" for volume "runner-xbb4ymon-project-50-concurrent-2-cache-3c3f060a0374fc8bc39395164f415a70": starting permission container: Error response from daemon: failed to start shim: fork/exec /usr/bin/containerd-shim: too many open files: unknown (linux_set.go:105:0s)
Will be retried in 3s ...
Using Docker executor with image registry.gitlab.systems.co.uk/devopstooling/dockerimages:node-14 ...
ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "jgfjdfkjdshufusdbksdbkjfbdjkbfjlsdnklfnsdklfnkls" for volume "runner-xbb4ymon-project-50-concurrent-2-cache-3c3f060a0374fc8bc39395164f415a70": starting permission container: Error response from daemon: failed to start shim: fork/exec /usr/bin/containerd-shim: too many open files: unknown (linux_set.go:105:0s)
Will be retried in 3s ...
Anyone has any idea what could be issue here and what can i do to fix this permanently.