Bazel not pulling from remote cache in CI when download_pkgs and install_pkgs used together

I've noticed when I run a Bazel rule that depends on download_pkgs and then install_pkgs into a Docker image that we aren't getting any cache hits from the remote cache even though in consecutive runs I can see the packages downloaded in the download_pkgs rule have the same hash.

I found this link from a while back which explains an issue whereby doing download_pkgs and then immediately install_pkgs can lead to non-deterministic builds but I didn't think this would happen when the hashes of the packages from download_pkgs were consistent.

I'm wondering whether anyone else has seen this issue and whether the workaround is like in the above link (push downloaded packages elsewhere as a tar and then use http_file to get them) or whether there is some fundamental doing Bazel in Docker with remote caching config I am missing?

example rules below:

download_pkgs(
    name = "download_ruby_apt_packages",
    packages = [
        "ca-certificates",
        "debsums",
        "g++",
        "git",
        "gnupg2",
        "libperconaserverclient20-dev",
        "libssl-dev",
        "make",
        "mysql-common",
        "percona-server-client-5.7",
        "percona-server-common-5.7",
        "ruby2.7",
        "ruby2.7-dev",
        "zlib1g-dev",
    ],
)


install_pkgs(
    name = "ubuntu2004_with_base_pkgs",
    image_tar = "@ubuntu2004//image",
    installables_tar = ":download_ruby_apt_packages.tar",
    installation_cleanup_commands = "rm -rf /var/lib/apt/lists/*",
    output_image_name = "ubuntu2004_with_base_pkgs",
)