In my current project we are using Jenkins with Docker-In-Docker based approach to build docker images.
I extended one of our jobs to use integration tests using docker-compose and Landoops Commandline testing tool coyote. I do build the production docker image in the build as well as docker images only used for integration testing.
After some while the docker space under (I think /var/lib/docker) fills up the whole disk and I get things like this:
mkdir /var/lib/docker/volumes/41d787304856fb065aa0d6342efe0a0c38672775fc044a3345ae5067ec8b8f3c: no space left on device
When I do a docker system prune together with cleaning volumes, the build/test runs again, but has to download a lot of docker images again, and this should not be done while another test is in the middle of downloading/building docker images.
So my conclusion is: Docker-In-Docker (especially local image repo and volume storage) is using the "outer" docker registry and volume storage.
My question is: Is there some solution to this problem without deleting all images/volumes after/before each job?
Update:
I did found this
"Device Mapper is not namespaced, so if multiple instances of Docker use it on the same machine, they will all be able to see (and affect) each other’s image and container backing devices. No bueno.There are workarounds for many of those issues; for instance, if you want to use AUFS in the inner Docker, just promote /var/lib/docker to be a volume and you’ll be fine. Docker added some basic namespacing to Device Mapper target names, so that if multiple invocations of Docker run on the same machine, they won’t step on each other."
from: https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
So it seems to me our OPS team simply mounted it without using devicemapper namespaces, so each job on the same node uses the same /var/lib/docker.