It looks like you're working on the same problem that I'm actively solving right now.
The big issue with the question is that container image builds with docker build are not deterministic or reproducible unless it happens to reuse the cache from a previous build. A container image build, even with the same filesystem layers, contains metadata on that build, and the metadata contains timestamps:
$ regctl manifest get localhost:5000/library/alpine --platform linux/amd64 --format body | jq .
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1472,
"digest": "sha256:0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2814559,
"digest": "sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139"
}
]
}
$ regctl blob get localhost:5000/library/alpine sha256:0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b | jq .
{
"architecture": "amd64",
"config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh"
],
"Image": "sha256:d49869997c508135352366cebd3509ee756bba1ceb8eef708a4c3ff0d481084a",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"container": "b714116bd3f3418e7b61a6d70dd7244382f0844e47a8d1d66dbf61cb1cb02b2b",
"container_config": {
"Hostname": "b714116bd3f3",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"/bin/sh\"]"
],
"Image": "sha256:d49869997c508135352366cebd3509ee756bba1ceb8eef708a4c3ff0d481084a",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"created": "2022-04-05T00:19:59.912662499Z",
"docker_version": "20.10.12",
"history": [
{
"created": "2022-04-05T00:19:59.790636867Z",
"created_by": "/bin/sh -c #(nop) ADD file:5d673d25da3a14ce1f6cf66e4c7fd4f4b85a3759a9d93efb3fd9ff852b5b56e4 in / "
},
{
"created": "2022-04-05T00:19:59.912662499Z",
"created_by": "/bin/sh -c #(nop) CMD [\"/bin/sh\"]",
"empty_layer": true
}
],
"os": "linux",
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:4fc242d58285699eca05db3cc7c7122a2b8e014d9481f323bd9277baacfa0628"
]
}
}
Both the "created" and "history" steps have timestamps that will be unique to the build. Changing those timestamps changes the digest of the config blob, which changes the digest of the image manifest.
The next issue you'll run into is that json serialization would need to be canonical. Some tools will use pretty formatting like jq, others will eliminate all unneeded whitespace for compactness, the order of listing multiple keys in a map doesn't need to be alphabetical, etc. So you need to ensure the same tool is always used for serialization and it has canonical output.
To build without pushing to a registry, you can have docker's buildkit output to an OCI layout tar file:
docker build --output type=oci,dest=/path/to/file.tar .
And in that tar, you will find an index.json with the digest of an image manifest as it was created by buildkit. I've been taking this a step further with regclient's image modification features, changing timestamps (in my case to the git commit time) and stripping other mutable values from the build. Then I verify the result matches a previous build.
Tools like cosign will allow you to sign an image using a digest rather than depending on the image in the registry, even before that image has been pushed.
The image mod feature in regclient is still very much a WIP, but you can see the current features here:
$ regctl image mod --help
EXPERIMENTAL: Applies requested modifications to an image
Usage:
regctl image mod <image_ref> [flags]
Flags:
--annotation stringArray set an annotation (name=value) (default )
--annotation-base stringArray set base image annotations (image/name:tag,sha256:digest) (default )
--buildarg-rm string delete a build arg (default "")
--buildarg-rm-regex string delete a build arg with a regex value (default "")
--config-time-max string max timestamp for a config (default "")
--create string Create tag
--data-max stringArray sets or removes descriptor data field (size in bytes) (default )
--expose-add stringArray add an exposed port (default )
--expose-rm stringArray delete an exposed port (default )
--external-urls-rm remove external url references from layers (first copy image with "--include-external") (default )
-h, --help help for mod
--label stringArray set an label (name=value) (default )
--label-to-annotation set annotations from labels (default )
--layer-rm-created-by string delete a layer based on history (created by string is a regex) (default "")
--layer-rm-index uint delete a layer from an image (index begins at 0) (default )
--layer-strip-file string delete a file or directory from all layers (default "")
--layer-time-max string max timestamp for a layer (default "")
--replace Replace tag (ignored when "create" is used)
--time-max string max timestamp for both the config and layers (default "")
--to-oci convert to OCI media types (default )
--volume-add stringArray add a volume definition (default )
--volume-rm stringArray delete a volume definition (default )
Global Flags:
--logopt stringArray Log options
--user-agent string Override user agent
-v, --verbosity string Log level (debug, info, warn, error, fatal, panic) (default "warning")
The other part of the puzzle is to make RUN steps reproducible. That's less trivial since not only do the files have timestamps, but the contents of the files being created could have timestamps or other mutable content, and the commands could pull from external mutable sources. Solving that part of the problem is still a work in progress for me.
