How to seperate COPY to multiple steps for caching in Dockerfile

Baekjun Kim 0
Viewed 22

I'm building an Docker image with big files (>1.0GB) and small python scripts. Big files are rarely changed, so I want to caching it.

The directory is looks like:

- app/
  - main.py
  - modules/
      - foo.py
      - bar.py
  - big_files/
      - bigone.tar
      - bigtwo.tar

My first Dockerfile:

FROM python3:latest
COPY ./app /opt/app

When I update python scripts, it have to COPY all files which consume a long time.

What I want to acheive:

FROM python3:latest
COPY ./app/big_files /opt/app/big_files
COPY ./app /opt/app

However, it also copy big files too.

How to COPY in two step for caching?

dockerfile docker
2 Answers

You should change your app's structure so that the big files are outside the /app folder.

If you don't want to do that, then you have to explicitly copy the files under /app:

FROM python3:latest
COPY ./app/big_files /opt/app/big_files
COPY ./app/*.py /opt/app/
COPY ./app/modules /opt/app/
useruGkL0

Each step is happening in its own layer. In your case that means

  1. Pull base image
  2. Copy Big Files
  3. Copy ./app

Everything must be run on the initial build.

With a setup similar to yours:

$ tree .
.
├── Dockerfile
└── app
    ├── bigFiles
    │   └── bigFile
    └── smallFile

and a dockerfile:

FROM python:latest

COPY ./app/bigFiles .
COPY ./app .

RUN echo 'Done'

Now if you run the exact same build again and nothing has changed, each layer can be reused:

$ docker build -t stackoverflowtest .

[+] Building 1.4s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 36B                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/python:latest                                                                                                                                                                                                                                                                                                                                                                       1.3s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                                                                                                                                                                                                                                                                                          0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 124B                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => [1/4] FROM docker.io/library/python:latest@sha256:e9c35537103a2801a30b15a77d4a56b35532c964489b125ec1ff24f3d5b53409                                                                                                                                                                                                                                                                                                                 0.0s
 => CACHED [2/4] COPY ./app/bigFiles .                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => CACHED [3/4] COPY ./app .                                                                                                                                                                                                                                                                                                                                                                                                          0.0s
 => CACHED [4/4] RUN echo 'Done'                                                                                                                                                                                                                                                                                                                                                                                                       0.0s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                                                                                0.0s
 => => writing image sha256:a78f5f730a893ccaaff102a7fe179461ec15c53bcbac926cb818e95d9d012875                                                                                                                                                                                                                                                                                                                                           0.0s
 => => naming to docker.io/library/stackoverflowtest

If you touch the big files, that layer has to be rebuilt:

$ echo "Change me" > app/bigFiles/bigFile
$ docker build -t stackoverflowtest .
[+] Building 0.9s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 36B                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/python:latest                                                                                                                                                                                                                                                                                                                                                                       0.6s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 146B                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => CACHED [1/4] FROM docker.io/library/python:latest@sha256:e9c35537103a2801a30b15a77d4a56b35532c964489b125ec1ff24f3d5b53409                                                                                                                                                                                                                                                                                                          0.0s
 => [2/4] COPY ./app/bigFiles .                                                                                                                                                                                                                                                                                                                                                                                                        0.0s
 => [3/4] COPY ./app .                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => [4/4] RUN echo 'Done'                                                                                                                                                                                                                                                                                                                                                                                                              0.2s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                                                                                0.0s
 => => writing image sha256:912022e447d8af0d20d77bc0ea41e0e1451012e1185e97e58f13409b6542bc19                                                                                                                                                                                                                                                                                                                                           0.0s
 => => naming to docker.io/library/stackoverflowtest

You can see that in the example above, the step 2 and 3 have changed because bigFiles is also part of app, so no layer could be cached.

However, if you only change the small files and not big files, the big file layer can be cached:

$ echo "Change me" > app/smallFile
$ docker build -t stackoverflowtest .
[+] Building 0.9s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 36B                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/python:latest                                                                                                                                                                                                                                                                                                                                                                       0.6s
 => [1/4] FROM docker.io/library/python:latest@sha256:e9c35537103a2801a30b15a77d4a56b35532c964489b125ec1ff24f3d5b53409                                                                                                                                                                                                                                                                                                                 0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 148B                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => CACHED [2/4] COPY ./app/bigFiles .                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => [3/4] COPY ./app .                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => [4/4] RUN echo 'Done'                                                                                                                                                                                                                                                                                                                                                                                                              0.1s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                                                                                0.0s
 => => writing image sha256:fdb687373689497d81bd6a14d4c30c6586b613b25463cdf5b387475700ccfb07                                                                                                                                                                                                                                                                                                                                           0.0s
 => => naming to docker.io/library/stackoverflowtest
Rick Rackow0
Related