How do I create a Go docker image with bazel when also needing libstdc++?

user_5860334 0
Viewed 1788

I am trying to create a docker image out of a mostly-go source base. The problem is that the go code also calls a C++ library (libsodium), so the generated container lacks libstdc++.

Here is my approach: first, I bundle the distroless go image with a few things created with pkg_tar():

container_image(
    name = "new_base_image",
    base = "@go_image_base//image",
    tars = [
      ":bash_static_tar",
      ":passwd_tar",
      ":dirs_tar",
    ],
    visibility = ["//:__subpackages__"],
)

Then I create my go image, package it, and push it to my registry:

go_image(
    name = "alice-go-image",
    embed = [":alice_lib"],
    base = "//:new_base_image",
    visibility = ["//visibility:public"],
)


container_image(
    name = "alice-image",
    base = ":alice-go-image",
    user = "alice",
    entrypoint = "/bin/sh",
    directory = "/opt/alice",
    workdir = "/opt/alice",
)

container_push(
    name = "alice-push",
    format = "Docker",
    image = ":alice-image",
    registry = "us-central1-docker.pkg.dev",
    repository = "mycoolproject/pub-docker/alice",
)

Unforunately, the Go code calls some vendored C++ code (libsodium, not that it matters), so attempting to run the resulting binary from within the container yields:

sh-5.1$ ./apollo                                                                                                                                                                        
./alice: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

I thought I could simply include the distroless CC image in a bundle, but I cannot figure out how to do it. I would think that putting

container_load(
    name = "distroless-cc",
    registry = "gcr.io",
    repository = "distroless/cc",
    tag = "latest",
)

in the WORKSPACE, then including the @distroless-cc//image label as one of the entries in the tars attribute of container_image would do the trick. Unfortunately, what it does instead is include the layer.tar files and associated metadata as part of the image! That is, if I change the first rule to:

container_image(
    name = "new_base_image",
    base = "@go_image_base//image",
    tars = [
      ":bash_static_tar",
      ":passwd_tar",
      ":dirs_tar",
      "@distroless-cc//image",
    ],
    visibility = ["//:__subpackages__"],
)

the result is:

drwxr-xr-x 0/0               0 1969-12-31 19:00 ./
drwxr-xr-x 0/0               0 1969-12-31 19:00 ./bin/
-r-xr-xr-x 0/0         1248656 1969-12-31 19:00 ./bin/sh
drwxr-xr-x 0/0               0 1969-12-31 19:00 ./etc/
-rw-r--r-- 0/0              74 1969-12-31 19:00 ./etc/passwd
drwxr-xr-x 0/0               0 1969-12-31 19:00 ./opt/
drwxr-xr-x 0/0               0 1969-12-31 19:00 ./opt/config/
-rw-r--r-- 0/0             776 1969-12-31 19:00 ./46bdf18e6b66057ca3b88eb8e0e2b8443297e36629e5fba3f33c2b2c717aa083.json
-rw-r--r-- 0/0         3061760 1969-12-31 19:00 ./4441c0604495fd6824901b4a57c65ffbb9025023edc1bee6f1a8fe2c312347a3/layer.tar
-rw-r--r-- 0/0             245 1969-12-31 19:00 ./4441c0604495fd6824901b4a57c65ffbb9025023edc1bee6f1a8fe2c312347a3/json
-rw-r--r-- 0/0               3 1969-12-31 19:00 ./4441c0604495fd6824901b4a57c65ffbb9025023edc1bee6f1a8fe2c312347a3/VERSION
-rw-r--r-- 0/0        17786880 1969-12-31 19:00 ./51aee51224937d5c4fe6e805be3dac78ace093d894d99923afb39a6fbcb9711b/layer.tar
-rw-r--r-- 0/0             321 1969-12-31 19:00 ./51aee51224937d5c4fe6e805be3dac78ace093d894d99923afb39a6fbcb9711b/json
-rw-r--r-- 0/0               3 1969-12-31 19:00 ./51aee51224937d5c4fe6e805be3dac78ace093d894d99923afb39a6fbcb9711b/VERSION
-rw-r--r-- 0/0         1996800 1969-12-31 19:00 ./605a66edf924fdfdddccc79e556050288e91ecf53a2d71fa8c17d793c7d26451/layer.tar
-rw-r--r-- 0/0             485 1969-12-31 19:00 ./605a66edf924fdfdddccc79e556050288e91ecf53a2d71fa8c17d793c7d26451/json
-rw-r--r-- 0/0               3 1969-12-31 19:00 ./605a66edf924fdfdddccc79e556050288e91ecf53a2d71fa8c17d793c7d26451/VERSION
-rw-r--r-- 0/0             359 1969-12-31 19:00 ./manifest.json
-rw-r--r-- 0/0              92 1969-12-31 19:00 ./repositories

Which is clearly not what I want!

Of course I could manually create an image by untarring the go and the cc distroless images together and then pushing it to my private registry, but that would require doing stuff manually, and I can't figure out how to do it with the bazel docker rules.

Help?

bazel docker go c++
2 Answers

You will need to assure that all of the dependancies for your code are installed in the image in order to be able to use the resulting image (without additional installation). This means that you need to install the libraries and binaries (in your case libstdc++ and other dependent libraries). Unfortunately, bazel rules do not run the Docker container, rather, they are assembling static files (layers) in a resulting image, so you will not be able to install them during the packaging.

One approach would be to create a base image with all tools and libraries you need and use this image as a starting point for the basel. This can be done with the standard docker build tools by creating a Dockerfile that prepares the environment and pushing the resulting file to the repository (or as a tar file) to be used as a base for basel

Another approach is to create the dependancies as a tar layers and use them to assemble the image with container_layer("tars"=[]) and add the layer to the container_image("layers"=), however this is not recommended due to two major issues: 1) Creating the tar is complex and prone to errors, and, 2) the tar is static, so the subsequent builds will not include the patches and minor updates.

user_28167030

As @JayEye later self answered in @jordanvrtanoski's answer, using @distroless-cc as base image should fix this problem.

To give more color on this, we first need to understand "distroless" images and how they are used by the various <language>_image rules from rules_docker.

Distroless Container Images

From GoogleContainerTools github page,

"Distroless" images contain only your application and its runtime dependencies. They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution.

The further list the various images available. Some examples include - gcr.io/distroless/[static-debian11 | base-debian11 | cc-debian11 | python3-debian11 | ...]

<language>_image rules

Now coming to the go_image rule, if you set the pure attribute of the rule to "on" then the static-debian11 image is used. If pure attribute is not set or set to "off" then the base-debian11 distroless image is used.

To see what each of these distroless container images contain, you can look at the distroless github page.

For example, the difference between base and static is specified here

This image contains a minimal Linux, glibc-based system.
It is intended for use directly by "mostly-statically compiled"
languages like Go, Rust or D.

Statically compiled applications (Go) that do not require
libc can use the gcr.io/distroless/static image, which contains:

ca-certificates
A /etc/passwd entry for a root user
A /tmp directory
tzdata
Most other applications (and Go apps that require libc/cgo) 
should start with gcr.io/distroless/base, which contains 
all of the packages in gcr.io/distroless/static, and

glibc
libssl
openssl

Lastly, the reason distroless-cc work is because it contains libgcc1 (and its dependencies) bundled into the image along with everything in the base image.

From documentation

the image contains everything in the base image, plus: libgcc1 and its dependencies

user_7716500
Related