When Dockerfile commands get cached

RUN apt-get update && apt-get install --no-install-recommends -yy curl

Unless a previous command already invalidated cache, this command should indeed re-use cache.

The first COPY command will use the cache only if requirements.txt is the same as before

COPY ./requirements.txt /opt/client_web/requirements.txt

Yes. Docker will use a checksum of this file to ensure this.

This line will cache only if the content of /opt/client_web/requirements.txt is exactly the same as before otherwise it will not use the cache.

RUN python3 -m pip install --default-timeout=100 -r /opt/client_web/requirements.txt

Not exactly, if command line change the cache won't be re-used. The cache will be used only if:

• The command string is exactly the same as before
• The cache was not invalidated by a previous command (such as COPY with a different file) - in your specific case a change to requirements.txt

Your statement is correct only if there is no instruction between this COPY and RUN. Another instruction placed between these two may invalidate cache.

Furthermore, Docker has no way to know that your command line actually uses the requirements.txt file, only a change in the command string may tell Docker to invalidate the cache on this specific step.

This will probably always cache and it's ok I don't mind. Of course I won't get a newer version of pip if it gets released:

RUN curl https://bootstrap.pypa.io/get-pip.py > get-pip.py && python3 get-pip.py

Yes, unless cache is invalidated by a previous instruction or command string changes.

If yes, we can say that the cache is not used only if the command involved files that changed.

No. COPY and ADD are only one of the ways Docker may invalidate cache. RUN may also invalidate cache, not because of file involved but because the string of the shell command changed. Additionally any Dockerfile instruction change may invalidate cache, even if not file is involved. See the Leverage build cache docs

I think the command are run again if:

• a previous layer changed (so a previous command or the from image)
• the docker context changed (like the .requirement.txt) this is valid for COPY & ADD and I think if you use the changes from the context in that command.
• the command itself changed.

The main issue I see there is that if you don't perform update, you main miss let say security fixes. Even if you use "latest" for your base image, docker will not download it again by default. So if you keep using the same build machine and use cache, you may really use the same version of everything for maybe several years and then miss a critical security fix.

I'd say when working locally in development and tuning the image, in particular working on the upper layers, the cache is extremely useful, you gain lot of time. But when creating the new official version of an image, you may prefer to be 100% sure it will update when you ask to update so you don't mix the security fixes.

Docker layers are created from top to bottom on your Dockerfile, once a layer gets rebuilt then all the following also will.

Let's take this example for a test drive:

from alpine

RUN apk add python3 py3-pip curl

COPY requirements.txt .

RUN pip install -r requirements.txt

ADD https://gist.githubusercontent.com/xr09/505962045b360227e4e74e8e72824006/raw/5b9e16e884ea85da535e7a0b9ae3c529415e7c59/more.txt more.txt

RUN pip install -r more.txt

RUN curl https://gist.githubusercontent.com/xr09/505962045b360227e4e74e8e72824006/raw/5b9e16e884ea85da535e7a0b9ae3c529415e7c59/more.txt --output new.txt

requirements.txt

flask
requests

more.txt on my gist

bottle

This is the build output:

$ docker build . -t alpi:1
Sending build context to Docker daemon  3.072kB
Step 1/7 : from alpine
 ---> a24bb4013296
Step 2/7 : RUN apk add python3 py3-pip curl
 ---> Using cache
 ---> bd355d28381d
Step 3/7 : COPY requirements.txt .
 ---> Using cache
 ---> ef21b82a5c36
Step 4/7 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> 5a9b0b218462
Step 5/7 : ADD https://gist.githubusercontent.com/xr09/505962045b360227e4e74e8e72824006/raw/5b9e16e884ea85da535e7a0b9ae3c529415e7c59/more.txt more.txt
Downloading [==================================================>]       6B/6B
 ---> Using cache
 ---> 3d1a8982086e
Step 6/7 : RUN pip install -r more.txt
 ---> Using cache
 ---> 60540c4a5745
Step 7/7 : RUN curl https://gist.githubusercontent.com/xr09/505962045b360227e4e74e8e72824006/raw/5b9e16e884ea85da535e7a0b9ae3c529415e7c59/more.txt --output new.txt
 ---> Using cache
 ---> dc4241b67a12
Successfully built dc4241b67a12
Successfully tagged alpi:1

These are the reasons when a layer in itself gets rebuilt, starting a "Domino effect" on bottom layers.

Layer 1 will always use the cache until the alpine image gets refreshed from Docker Hub.

Layer 2 only gets rebuilt when you modify the line.

Layer 3 rebuilds when requirements.txt changes.

Layer 4 only triggers a rebuild when the RUN command is changed. (or when upper layer did of course)

Layer 5 is a tricky one, but looks like Docker is downloading the file every time and checking for changes. In this example it downloaded the file but still used the cached layer. This issue will provide some insight.

Layer 6 is the same as layer 4.

Layer 7 will always use the cache unless upper layers or the RUN command itself changed.

Here's a nice article on this topic: https://pythonspeed.com/articles/docker-caching-model/