Getting reproducible docker layers on different hosts

Problem: I can't reproduce docker layers using exactly same content (on one machine or in CI cluster where something is built from git repo)

Consider this simple example

$ echo "test file" > test.txt
$ cat > Dockerfile <<EOF
FROM alpine:3.8

COPY test.txt /test.txt
EOF

If I build image on one machine with caching enabled, then last layer with copied file would be shared across images

$ docker build -t test:1 .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM alpine:3.8
3.8: Pulling from library/alpine
cd784148e348: Already exists
Digest: sha256:46e71df1e5191ab8b8034c5189e325258ec44ea739bba1e5645cff83c9048ff1
Status: Downloaded newer image for alpine:3.8
 ---> 3f53bb00af94
Step 2/2 : COPY test.txt /test.txt
 ---> decab6a3fbe3
Successfully built decab6a3fbe3
Successfully tagged test:1

$ docker build -t test:2 .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM alpine:3.8
 ---> 3f53bb00af94
Step 2/2 : COPY test.txt /test.txt
 ---> Using cache
 ---> decab6a3fbe3
Successfully built decab6a3fbe3
Successfully tagged test:2

But with cache disabled (or simply using another machine) I got different hash values.

$ docker build -t test:3 --no-cache .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM alpine:3.8
 ---> 3f53bb00af94
Step 2/2 : COPY test.txt /test.txt
 ---> ced4dff22d62
Successfully built ced4dff22d62
Successfully tagged test:3

At the same time history command shows that file content was same

$ docker history test:1
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
decab6a3fbe3        6 minutes ago       /bin/sh -c #(nop) COPY file:d9210c40895e

$ docker history test:3
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
ced4dff22d62        27 seconds ago      /bin/sh -c #(nop) COPY file:d9210c40895e

Am I missing something or this behavior is by design?

Are there any technics to get reproducible/reusable layers that does not force me to do one of the following

1. Share docker cache across machines
2. Do a pull of "previous" image before building next

Ultimately this problem prevents me from getting thin layers with constantly changing app code while keeping layers of my dependencies in separate and infrequently changed layer.