I have what feels like a fairly standard ECS deployment:
- My cluster is backed by two EC2 instances, both running CoreOS with the ECS agent
- The cluster has a handful of services, each of which has associated tasks
- Each service is running one task at a time – usually the latest task definition
- The container images used in my task definitions are hosted in ECR
Several times now I’ve gone to deploy a new instance of a service, only to have ECS get unhappy because it’s unable to find space on my EC2 instance for the new image – I get “No space left on device” errors in CloudWatch.
If I SSH into the instance, I find that it’s got an old copy of all my Docker images and containers – almost 100 of the former, 500 of the latter. I can free up disk space by running the docker rm and docker rmi commands manually, but I naively assumed this was the sort of thing ECS would manage for me.
When I’m setting up services and tasks, I never usually interact with the individual hosts or containers – I trust ECS to schedule containers for me. I expected it would also purge old container images to free up disk space as required. (I’m aware this isn’t functionality provided by Docker.)
I could set alarms to monitor disk space on my EC2 instances, and log on to free disk space as required – but I’m wondering if I’ve missed something.
Can I tell ECS to purge old/unused container [images] from my EC2 instances?