How to debug docker pull layer by layer?

user_4440766 0
Viewed 5026

[situation] So we have this docker image. Like all other images, it consists of a couple of layers. Once we have spotted that during docker pull image:v42, two specific layers take a very long time to download, at some point they even hung up for a couple of minutes.

Somehow I'd like to debug and find out which part of the Dockerfile is in those two layers. After executing docker history image-id and even by using Dive, I have found that layers digests are completely different from the digest shown in Terminal during docker pull. Even the sizes of the layers are different.

[question] Is there a way to correlate those "docker history" digests with "docker pull" digests? Or i.e. how can I find out what is downloaded in those layers?

[p.s.]The only thing that comes to my mind and might help - is to comment out the whole Dockerfile, start uncommenting line by line and build an image with every new line. In the end, I will stumble over those layers. But there must be a better way. No?

UPDATE (Sept. 12 2020 | Docker 19.03.12): we have found out that running Docker in rootless mode doubles the pull times. i.e. 5GB image docker pull as root --> ~4min. and docker pull in rootless mode --> ~9-10min. The big chunk of that time goes to downloading those 2 layers and extracting all the layers.

dockerfile docker
2 Answers

It will be difficult to specify why the layers are slow other than just their size. If there are other reasons upstream, it's a lot of moving pieces from Cloudflare to Docker's hosting (probably in something like an AWS S3 bucket) and then there's your local network. There are also rate limits that kick in for very heavy usage (those limits will become more visible in another few months for other users). Assuming this is just a large layer, you can pull the blob directly from the registry to inspect it. Here's a couple scripts to hit the registry API with curl:

$ cat manifest-v2.sh 
#!/bin/sh

ref="${1:-library/ubuntu:latest}"
repo="${ref%:*}"
tag="${ref##*:}"
api="application/vnd.docker.distribution.manifest.v2+json"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
        | jq -r '.token')
curl -H "Accept: ${api}" \
     -H "Authorization: Bearer $token" \
     -s "https://registry-1.docker.io/v2/${repo}/manifests/${tag}" | jq .


$ cat get-blob.sh 
#!/bin/sh

ref="${1:-library/ubuntu:latest}"
repo="${ref%:*}"
tag="${ref##*:}"
digest="$2"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
        | jq -r '.token')
curl -H "Authorization: Bearer $token" \
     -s -L -o - "https://registry-1.docker.io/v2/${repo}/blobs/${digest}"

And running those, you can then see the digests you were looking at in the pull and download that specific layer, run it through tar, and see what's in the layer:

$ ./manifest-v2.sh weberstephanhd/iacbox2:v380
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 36642,
    "digest": "sha256:4588fe154684dfe09b266e2a122b0789dc7ee89ff284fb140f14962fa2d5c754"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 26697127,
      "digest": "sha256:7595c8c21622ea8a8b9778972e26dbbe063f7a1c4b0a28a80a34ebb3d343b586"
    },
... lots of other layers ...
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 19899,
      "digest": "sha256:5350cb37a04f91a25c1e2f288310f58769cf77761c00e68059931ec3dcc67301"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1364512,
      "digest": "sha256:476cd410766a5218a41bec1d3c772653ced6d5ad0cc14e52c303901da0255327"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 22698271,
      "digest": "sha256:1437ee729887a466d9d51b97321ce22f7f15c9340a622ce876f90cd7c0bc5952"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 17743524,
      "digest": "sha256:c7ed8287f08ff5f00a8c6a29afb9ceaa0bcf0e2fa0191f7bbb41aa8213ed55f6"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 5661386,
      "digest": "sha256:2e8965a34b4adf2d08f2cf12715ee40d1872c82ebda0fd1fb3711510b10ad07d"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 129,
      "digest": "sha256:6e71916e2b6e0112ea835e239c5770c1954f53c33d8dc80e0285af4b562a07e7"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 115,
      "digest": "sha256:b9677e754d0339eca6ffe9889d7d920f69efeb9987eb75308ee0479d11fa30cc"
    }
  ]
}

$ ./get-blob.sh weberstephanhd/iacbox2:v380 sha256:5350cb37a04f91a25c1e2f288310f58769cf77761c00e68059931ec3dcc67301 | tar -tzvf -
drwxr-xr-x root/root         0 2020-09-04 07:49 etc/
drwxr-xr-x root/root         0 2020-09-04 07:49 etc/profile.d/
-rw-r--r-- root/root      1186 2020-09-04 07:49 etc/profile.d/iacbox-env.sh
drwxr-xr-x root/root         0 2020-09-04 07:49 etc/skel/
drwxr-xr-x root/root         0 2020-09-04 07:49 etc/skel/.azure/
-rwxr-xr-x root/root         0 1969-12-31 19:00 etc/skel/.azure/.wh..wh..opq
-rw-r--r-- root/root        25 2020-09-04 07:49 etc/skel/.azure/telemetry.json
-rw-r--r-- root/root       167 2020-09-04 07:49 etc/skel/.bash_aliases
-rw-r--r-- root/root       169 2020-09-04 07:49 etc/skel/.rvmrc
drwx------ root/root         0 2020-09-04 07:49 root/
drwxr-xr-x root/root         0 2020-09-04 07:49 root/.azure/
-rwxr-xr-x root/root         0 1969-12-31 19:00 root/.azure/.wh..wh..opq
-rw-r--r-- root/root        25 2020-09-04 07:49 root/.azure/telemetry.json
-rw-r--r-- root/root       167 2020-09-04 07:49 root/.bash_aliases
-rw-r--r-- root/root       169 2020-09-04 07:49 root/.rvmrc
drwxr-xr-x root/root         0 2020-09-04 07:49 root/.ssh/
-rwxr-xr-x root/root         0 1969-12-31 19:00 root/.ssh/.wh..wh..opq
-rw-r--r-- root/root        51 2020-09-04 07:49 root/.ssh/config
-rwxr-xr-x root/root      1249 2020-09-04 07:49 root/profile.d_iacbox-env.sh
drwxrwxrwt root/root         0 2020-09-04 07:49 tmp/
-rwxr-xr-x root/root      2288 2020-09-04 07:49 tmp/acceptance-test.sh
-rw------- root/root         0 2020-09-04 07:49 tmp/.wh.config_files
drwxr-xr-x root/root         0 2020-07-13 10:48 usr/
drwxr-xr-x root/root         0 2020-09-04 07:47 usr/local/
drwxr-xr-x 503/staff         0 2020-09-04 07:49 usr/local/bin/
lrwxrwxrwx root/root         0 2020-09-04 07:49 usr/local/bin/bosh2 -> /usr/local/bin/bosh
-rwxr-xr-x root/root     15672 2020-09-04 07:49 usr/local/bin/execute-iac-command.sh
-rwxr-xr-x root/root     16675 2020-09-04 07:49 usr/local/bin/iac-cmd.sh
-rwxr-xr-x root/root      1334 2020-09-04 07:49 usr/local/bin/iac-validate.sh
-rwxr-xr-x root/root      3850 2020-09-04 07:49 usr/local/bin/prepare_git_credentials.sh
-rwxr-xr-x root/root        47 2020-09-04 07:49 usr/local/bin/print-iaas-cli-environment.sh
-rwxr-xr-x root/root       111 2020-09-04 07:49 usr/local/bin/sshtunnel-to-landscape.sh
-rwxr-xr-x root/root     15700 2020-09-04 07:49 usr/local/bin/tunnel-to-landscape.sh
-rwxr-xr-x root/root       824 2020-09-04 07:49 usr/local/bin/whitelist-check-external-ips.sh
-rwxr-xr-x root/root       536 2020-09-04 07:49 usr/local/bin/whitelist-create-external-ips.sh
-rwxr-xr-x root/root       535 2020-09-04 07:49 usr/local/bin/whitelist-import-external-ips.sh

As for why the digests are different, that comes down to compression. The media type for these blobs is a compressed tar on the registry, but once downloaded to the local docker engine, it uncompresses the layer and the local digest is on the uncompressed tar. That also explains why the sizes don't match.

To align the layers with the history, you can compare the history with the manifest, just be careful of the order of the lines (docker history shows lines in reverse as does the v1 registry API, while the v2 API calls will show the lines from oldest to newest) and not every line in the history results in a new blob. That becomes a bit more apparent if you look at the config json associated with an image:

$ more get-config-v2.sh
#!/bin/sh

ref="${1:-library/ubuntu:latest}"
repo="${ref%:*}"
tag="${ref##*:}"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
        | jq -r '.token')
digest=$(curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
              -H "Authorization: Bearer $token" \
              -s "https://registry-1.docker.io/v2/${repo}/manifests/${tag}" \
         | jq -r .config.digest)
curl -H "Accept: application/vnd.docker.container.image.v1+json" \
     -H "Authorization: Bearer $token" \
     -s -L "https://registry-1.docker.io/v2/${repo}/blobs/${digest}" | jq .

$ ./get-config-v2.sh weberstephanhd/iacbox2:v380
{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "root",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/opt/ostoolset/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin",
      "CF_HOME=/root",
      "CF_PLUGIN_HOME=/usr/local/lib/cf_plugins",
      "TF_PLUGIN_DIR=/usr/local/lib/terraform_plugins",
      "GOROOT=/usr/local/go",
      "PYTHONWARNINGS=ignore:Certificate for :::",
      "OS_VOLUME_API_VERSION=2",
      "LANG=en_US.UTF-8",
      "LANGUAGE=en_US.UTF-8",
      "LC_ALL=en_US.UTF-8"
    ],
    "Cmd": [
      "/bin/bash"
    ],
    "ArgsEscaped": true,
    "Image": "sha256:622904aeb073d26c0a17d87bf4df9cb026ffa99f1f5e9f3ff0fb6135e6aeb54e",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": null
  },
  "container": "af499c54916b436d2fee37da88a6e48ee1e1b218dece9747773396567cfcab23",
  "container_config": {
    "Hostname": "af499c54916b",
    "Domainname": "",
    "User": "root",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/opt/ostoolset/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin",
      "CF_HOME=/root",
      "CF_PLUGIN_HOME=/usr/local/lib/cf_plugins",
      "TF_PLUGIN_DIR=/usr/local/lib/terraform_plugins",
      "GOROOT=/usr/local/go",
      "PYTHONWARNINGS=ignore:Certificate for :::",
      "OS_VOLUME_API_VERSION=2",
      "LANG=en_US.UTF-8",
      "LANGUAGE=en_US.UTF-8",
      "LC_ALL=en_US.UTF-8"
    ],
    "Cmd": [
      "/bin/sh",
      "-c",
      "#(nop) ",
      "USER root"
    ],
    "ArgsEscaped": true,
    "Image": "sha256:622904aeb073d26c0a17d87bf4df9cb026ffa99f1f5e9f3ff0fb6135e6aeb54e",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {}
  },
  "created": "2020-09-04T11:49:53.021161021Z",
  "docker_version": "19.03.6",
  "history": [
    {
      "created": "2020-07-24T14:38:19.482143079Z",
      "created_by": "/bin/sh -c #(nop) ADD file:7d9bbf45a5b2510d44d3206a028cf6502757884d49e46d3d2e6356c3a92c4309 in / "
    },
    {
      "created": "2020-07-24T14:38:20.335965442Z",
      "created_by": "/bin/sh -c [ -z \"$(apt-get indextargets)\" ]"
    },
    {
      "created": "2020-07-24T14:38:21.071294363Z",
      "created_by": "/bin/sh -c set -xe \t\t&& echo '#!/bin/sh' > /usr/sbin/policy-rc.d \t&& echo 'exit 101' >> /usr/sbin/policy-rc.d \t&& chmod +x /usr/sbin/policy-rc.d \t\t&& dpkg-divert --local --rename --add /sbin/initctl \t&& cp -a /usr/sbin/policy-rc.d /sbin/initctl \t&& sed -i 's/^exit.*/exit 0/' /sbin/initctl \t\t&& echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/docker-apt-speedup \t\t&& echo 'DPkg::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' > /etc/apt/apt.conf.d/docker-clean \t&& echo 'APT::Update::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' >> /etc/apt/apt.conf.d/docker-clean \t&& echo 'Dir::Cache::pkgcache \"\"; Dir::Cache::srcpkgcache \"\";' >> /etc/apt/apt.conf.d/docker-clean \t\t&& echo 'Acquire::Languages \"none\";' > /etc/apt/apt.conf.d/docker-no-languages \t\t&& echo 'Acquire::GzipIndexes \"true\"; Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/docker-gzip-indexes \t\t&& echo 'Apt::AutoRemove::SuggestsImportant \"false\";' > /etc/apt/apt.conf.d/docker-autoremove-suggests"
    },
    {
      "created": "2020-07-24T14:38:21.85928744Z",
      "created_by": "/bin/sh -c mkdir -p /run/systemd && echo 'docker' > /run/systemd/container"
    },
    {
      "created": "2020-07-24T14:38:22.027273323Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:28.397707805Z",
      "created_by": "/bin/sh -c #(nop)  ENV CF_HOME=/root",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:28.50996911Z",
      "created_by": "/bin/sh -c #(nop)  ENV CF_PLUGIN_HOME=/usr/local/lib/cf_plugins",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:28.613680523Z",
      "created_by": "/bin/sh -c #(nop)  ENV TF_PLUGIN_DIR=/usr/local/lib/terraform_plugins",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:28.716721682Z",
      "created_by": "/bin/sh -c #(nop)  ENV GOROOT=/usr/local/go",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:28.819392564Z",
      "created_by": "/bin/sh -c #(nop)  ENV PATH=/opt/ostoolset/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:28.92341478Z",
      "created_by": "/bin/sh -c #(nop)  ENV PYTHONWARNINGS=ignore:Certificate for :::",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:29.024894545Z",
      "created_by": "/bin/sh -c #(nop)  ENV OS_VOLUME_API_VERSION=2",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:29.618098106Z",
      "created_by": "/bin/sh -c rm /bin/sh && ln -sf /bin/bash /bin/sh"
    },
    {
      "created": "2020-08-27T04:23:40.330900434Z",
      "created_by": "/bin/sh -c apt-get update &&     DEBIAN_FRONTEND=noninteractive     apt-get install -y     apt-utils     apt-transport-https     gnupg2     curl     wget     locales &&     rm -rf /var/lib/apt/lists/*"
    },
    {
      "created": "2020-08-27T04:23:41.633234144Z",
      "created_by": "/bin/sh -c locale-gen \"en_US.UTF-8\""
    },
    {
      "created": "2020-08-27T04:23:41.742325545Z",
      "created_by": "/bin/sh -c #(nop)  ENV LANG=en_US.UTF-8",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:41.845327099Z",
      "created_by": "/bin/sh -c #(nop)  ENV LANGUAGE=en_US.UTF-8",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:41.954629412Z",
      "created_by": "/bin/sh -c #(nop)  ENV LC_ALL=en_US.UTF-8",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T04:23:42.08703115Z",
      "created_by": "/bin/sh -c #(nop) COPY file:298c66e94d89d7e189c54b0e79c24ac300dd4f3524469d0d8adcf16d5f3f5776 in /etc/apt/sources.list.d/extended.list "
    },
    {
      "created": "2020-08-27T04:23:45.007887008Z",
      "created_by": "/bin/sh -c curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - &&     curl -sSL https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - &&     curl -L https://packages.microsoft.com/keys/microsoft.asc | apt-key add - &&     curl -sSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - &&     curl -sSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - &&     mkdir -p \"/usr/local/share/ca-certificates/sap.com\" \"/usr/local/share/ca-certificates/verizon\" &&     curl -sSL http://aia.pki.co.sap.com/aia/SAPNetCA_G2.crt -o \"/usr/local/share/ca-certificates/sap.com/SAPNetCA_G2.crt\" &&     curl -sSL http://aia.pki.co.sap.com/aia/SAP%20Global%20Root%20CA.crt -o \"/usr/local/share/ca-certificates/sap.com/SAP_Global_Root_CA.crt\" &&     curl -sSL https://de.ssl-tools.net/certificates/f326e9f894088fb560a001aa2c0ea8b1c20e6c35.pem -o \"/usr/local/share/ca-certificates/verizon/Verizon_Public_SureServer_CA_G14-SHA2.crt\" &&     update-ca-certificates"
    },
    {
      "created": "2020-08-27T04:23:45.871796489Z",
      "created_by": "/bin/sh -c wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -"
    },
    {
      "created": "2020-08-27T04:23:46.464366636Z",
      "created_by": "/bin/sh -c echo \"deb http://apt.postgresql.org/pub/repos/apt/ bionic\"-pgdg main | tee  /etc/apt/sources.list.d/pgdg.list"
    },
... bunch of history trimmed ...
    {
      "created": "2020-09-04T11:49:51.582086565Z",
      "created_by": "|17 TF_BIN_RELEASE_URL_PREFIX=https://releases.hashicorp.com/terraform TF_PROVIDER_VERSION_ALI=1.70.0 TF_PROVIDER_VERSION_ARCHIVE=1.2.2 TF_PROVIDER_VERSION_AWS=2.70.0 TF_PROVIDER_VERSION_AZURE=2.17.0 TF_PROVIDER_VERSION_AZUREAD=0.10.0 TF_PROVIDER_VERSION_AZURE_LEGACY=1.44.0 TF_PROVIDER_VERSION_DNS=2.1.1 TF_PROVIDER_VERSION_GOOGLE=3.15.0 TF_PROVIDER_VERSION_LOCAL=1.4.0 TF_PROVIDER_VERSION_NULL=2.1.2 TF_PROVIDER_VERSION_OPEN_STACK=1.20.0 TF_PROVIDER_VERSION_POSTGRESQL=1.1.0 TF_PROVIDER_VERSION_RANDOM=2.3.0 TF_PROVIDER_VERSION_TEMPLATE=2.1.2 TF_VERSION=0.12.28 VAULT_VERSION=1.4.2 /bin/sh -c cp /root/go/bin/ossutil /usr/local/bin"
    },
    {
      "created": "2020-09-04T11:49:51.726671638Z",
      "created_by": "/bin/sh -c #(nop)  ARG VERSION=latest",
      "empty_layer": true
    },
    {
      "created": "2020-09-04T11:49:52.311382499Z",
      "created_by": "|18 TF_BIN_RELEASE_URL_PREFIX=https://releases.hashicorp.com/terraform TF_PROVIDER_VERSION_ALI=1.70.0 TF_PROVIDER_VERSION_ARCHIVE=1.2.2 TF_PROVIDER_VERSION_AWS=2.70.0 TF_PROVIDER_VERSION_AZURE=2.17.0 TF_PROVIDER_VERSION_AZUREAD=0.10.0 TF_PROVIDER_VERSION_AZURE_LEGACY=1.44.0 TF_PROVIDER_VERSION_DNS=2.1.1 TF_PROVIDER_VERSION_GOOGLE=3.15.0 TF_PROVIDER_VERSION_LOCAL=1.4.0 TF_PROVIDER_VERSION_NULL=2.1.2 TF_PROVIDER_VERSION_OPEN_STACK=1.20.0 TF_PROVIDER_VERSION_POSTGRESQL=1.1.0 TF_PROVIDER_VERSION_RANDOM=2.3.0 TF_PROVIDER_VERSION_TEMPLATE=2.1.2 TF_VERSION=0.12.28 VAULT_VERSION=1.4.2 VERSION=v380 /bin/sh -c echo ${VERSION} > /docker_image_version"
    },
    {
      "created": "2020-09-04T11:49:52.91091882Z",
      "created_by": "|18 TF_BIN_RELEASE_URL_PREFIX=https://releases.hashicorp.com/terraform TF_PROVIDER_VERSION_ALI=1.70.0 TF_PROVIDER_VERSION_ARCHIVE=1.2.2 TF_PROVIDER_VERSION_AWS=2.70.0 TF_PROVIDER_VERSION_AZURE=2.17.0 TF_PROVIDER_VERSION_AZUREAD=0.10.0 TF_PROVIDER_VERSION_AZURE_LEGACY=1.44.0 TF_PROVIDER_VERSION_DNS=2.1.1 TF_PROVIDER_VERSION_GOOGLE=3.15.0 TF_PROVIDER_VERSION_LOCAL=1.4.0 TF_PROVIDER_VERSION_NULL=2.1.2 TF_PROVIDER_VERSION_OPEN_STACK=1.20.0 TF_PROVIDER_VERSION_POSTGRESQL=1.1.0 TF_PROVIDER_VERSION_RANDOM=2.3.0 TF_PROVIDER_VERSION_TEMPLATE=2.1.2 TF_VERSION=0.12.28 VAULT_VERSION=1.4.2 VERSION=v380 /bin/sh -c echo migrated > /migrated"
    },
    {
      "created": "2020-09-04T11:49:53.021161021Z",
      "created_by": "/bin/sh -c #(nop)  USER root",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:7ef3687765828a9cb2645925f27febbac21a5adece69e8437c26184a897b6ec7",
      "sha256:83f4287e1f0496940f8c222ca09cbaf2c7f564a10c57b4609800babe8d1b5b32",
      "sha256:d3a6da143c913c5e605737a9d974638f75451b5c593e58eb7e132fcf0e23c6db",
      "sha256:8682f9a74649fb9fc5d14f827a35259aae8b58c57df8d369f6aa2e92865930c2",
      "sha256:6fd2b828f15530b3e91fdc47694d1165a90a2e33edce9c30961a1fb0c92f2d51",
      "sha256:c41b334c1d5f7c94942e7b6372d5c18f43e8840eb12d7683f6b45856d2776d1a",
      "sha256:a180f973b01e39cd5eb3fc04222d06be573eb308e5a096157c3a9c5c523e3e0c",
      "sha256:424841a53760a9261eb7b4d3dd7c70b62bc74525a13412d0dbbca8deea4255d3",
      "sha256:198605a838ddf053e6120869caebef0272b725c354a42a1d20beeaf8e02674c0",
      "sha256:487242c09dbb7034f6f38585f74f28255d0a574c3db706c830e8408d2436b2fe",
      "sha256:68b89d709fcb495c984b811f195c86dc36831f2cd4f95f38a26fed3baa4e2c5e",
      "sha256:877656568034f2328abd9d73036463d160b0da912ace2535e5c13a75c893413f",
      "sha256:70f7e4f7c12ea90fda9f0442bbb4e6ea7bae448b56d8b6c83c365adb09254367",
      "sha256:0576dc0190175293ca7cbba2a6ea93957db46458246da26f1434b432d66173db",
      "sha256:218cf40a20bbf60704152566772bc0bf803374c594e47261f05ab2b207f497fb",
      "sha256:f5845a9ea4efd834b747a109209d46b6dcb5e05e88a27df67de4aba03dd92f18",
      "sha256:000cddfa5fea51b517db796168add34b9648955dfa6421c8c158709d9687cb32",
      "sha256:4bd3750abf95ceb7d4e6c8f47acdc1c9db754325e050776186ae0a681e032fcd",
      "sha256:810c4b2cb213baf64895dec8566410e7b401436bf524c33d55893a45748c1bba",
      "sha256:5dcb620bac21a7449919d730e5e7ed23f9a0d791f72d3709b5cf3e81853adcd1",
      "sha256:09af1a244f314fe3558afa8d42e66ac376a18bf010544ba4872f3fe7461d1382",
      "sha256:c16ddbd687d181a095ca3ec8b108cf0d27a7a5408cc6e5073c03c903fbd344d3",
      "sha256:8b77549540c08a2571a7d7e8d11cc59519f36b613c0e1ef3c5d713d39f4e3507",
      "sha256:01079aef52a4ee46973ddd207c40f7b1ff4688216a5a05bfb37feea2041fc214",
      "sha256:2e43dfce6411894ef682d3ba313021264700033b0789f826b1304c8874695bc9",
      "sha256:616a5dfd09e5305cc271f77727a84d0c28628c758a994df65df75c33444ea42a",
      "sha256:8c9c8ab8442e9a7e3094090ac8d6659acb050d477d306841cea37eef4a379eed",
      "sha256:047be9dbcd6162c17cdfaee5380611072e27fb7b3de1c462d01d63f436b04f44",
      "sha256:8d25d3692cf4d95dca0327da7d8632c27cbdee71110a9c78628eba94fdc04e6d",
      "sha256:b21e2e4a55c8861e36503e6bb7385968e7db49247e07772927c3baf5c06336de",
      "sha256:ab0696412395709d6a71e21bba15cc474a52a34798149cee761d3accbb896296",
      "sha256:c81651668c205be10467a540129bd91eca20711f22b929a94839f780b1ff0d3f",
      "sha256:57cdc9e21eff8e78820f5222b3e21a06808564aa19884fa3003b12f58b325c7f",
      "sha256:18d72968405bb4ea92c826a718b080cc8b2f0d310f9c791af0bcf6cfd4e9dfc7",
      "sha256:1efa41d22392b374475e2472d0f6c710498052330f5c218296fbf327303d4890",
      "sha256:dd2dd93244de62000c96955f424a55699c6458d64932e4b170e0c42d095b3bc0",
      "sha256:432d521b81b9bcaef07786922cc16581cc6f008ea2bea79fd39d01a496814599"
    ]
  }
}

In there, you'll see "empty_layer": true which indicates that step of the Dockerfile only produced a change to the config.json and didn't create a new filesystem layer. So if you're careful you can count the layers in the history and in the manifest to identify which history command resulted in which layer/blob.

P.S. looking at the command history in those layers, I expect some of the large layers are because of terraform, the binaries and plugins are not small (I've fought with this in the past trying to make a quick provisioner container only to find my image is often larger than what I wanted to provision).

user_5962850

You can try dive, which is a a docker layer inspector.

user_4139240
Related