Does ECS use Docker layer cache to speed-up deployments?

user_550208 0
Viewed 1096

Imagine a typical ECS scenario:

  1. I deploy task definition 1 which uses MYIMAGE:1.
  2. Later I deploy task definition 2 which uses MYIMAGE:2.

The second deployment goes to the same EC2 hosts as the first one. So theoretically ECS could use Docker layer cache when pulling MYIMAGE:2 from a container registry (this would probably result in a much faster pull).

But does it use it in practice? Is there a way to enable it? Documentation does not seem to mention it.

amazon-ecs amazon-web-services
2 Answers

Is ecs-agent using the standard docker pull behaviour?

Since the question was refined in the comments above, I try to answer whether the ecs-agent is using the standard docker pull mechanics.

  • We know from the AWS docs that the configuration parameter is called ECS_IMAGE_PULL_BEHAVIOR. [1]
  • AWS ECS uses the ecs-agent on instances to manage containers. Since the tool is open-source, we have to do the work and look for the code which pulls the images. It is hosted on GitHub. [2]
  • If we search for the PR which introduced the pull behaviour switch, we find the following: #1840.
  • From there on, we see that the docker_task_engine is the starting point. Then, the docker_client is called which in turn uses the go-docker project to send an image_create request.
  • We can look at the docker engine API to see that this is a standard image pull.

Conclusion

You can customize the ECS docker pull behaviour by telling the ecs-agent when it should fire off a pull request and whether automated image cleanup should be enabled or not. When ecs-agent attempts to pull an image from the container registry, it uses the standard docker "create image" request [3].

Finally the docker engine requests a standard image pull from the docker daemon. [4]
If you set the ecs-agent environment variable ECS_IMAGE_PULL_BEHAVIOR to prefer-cached, MYIMAGE:1 is pulled entirely the first time it is used. Subsequently, the same EC2 instance does a docker pull to retrieve MYIMAGE:2.

I tried to figure out how PullImage in docker daemon works under the hood, but I could not find any useful information. I guess it does a delta update of the layers which changed between MYIMAGE:1 and MYIMAGE:2 (if MYIMAGE:1 is still available - which requires automated image cleanup to be turned off), but I cannot fully confirm it...

The docker FAQs [5] state the following:

Versioning. [...] Docker also implements incremental uploads and downloads, similar to git pull, so new versions of a container can be transferred by only sending diffs.

References

[1] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html
[2] https://github.com/aws/amazon-ecs-agent
[3] https://docs.docker.com/engine/api/v1.24/#create-an-image
[4] https://github.com/docker/engine/blob/master/api/server/router/image/image_routes.go#L78
[5] https://docs.docker.com/engine/faq/

user_104734690

ecs issues docker pull command on MYIMAGE:1. if your MYIMAGE:2 has same layers of MYIMAGE:1 then docker will use the locally available image layers and just docker pull on MYIMAGE:2 will only pull the metadata not the actual layers.

detailed doc. https://docs.docker.com/engine/reference/commandline/pull/

user_48503770
Related