How to stop `gcloud component update` from keeping a backup?

user_3213981 0
Viewed 953

I am building a google-cloud-sdk base-image and I am trying to keep it lean.

ARG GCLOUD_SDK_VERSION=285.0.1-alpine
FROM google/cloud-sdk:$GCLOUD_SDK_VERSION

# Install Java 8 for Datastore emulator
RUN apk add --update --no-cache \
        openjdk8-jre
RUN gcloud components install \
        cloud-datastore-emulator \
        pubsub-emulator \
        beta \
        --quiet
...

So far everything is fine. And when I build and look at the output of the gcloud components install command I feel confident:

┌──────────────────────────────────────────────────┐
│       These components will be installed.        │
├──────────────────────────┬────────────┬──────────┤
│           Name           │  Version   │   Size   │
├──────────────────────────┼────────────┼──────────┤
│ Cloud Datastore Emulator │      2.1.0 │ 18.4 MiB │
│ Cloud Pub/Sub Emulator   │ 2019.09.27 │ 34.9 MiB │
│ gcloud Beta Commands     │ 2019.05.17 │  < 1 MiB │
└──────────────────────────┴────────────┴──────────┘

My final image size, however, is shocking: 1.11GB. When I look at docker history I see that the component installation actually contributed 654MB to the final image size:

CREATED BY                                      SIZE
/bin/sh -c gcloud components install        …   654MB
/bin/sh -c apk add --update --no-cache      …   78.2MB

I am currently assuming it has something to do with the final line of the installation process.

╔════════════════════════════════════════════════════════════╗
╠═ Creating backup and activating new installation          ═╣
╚════════════════════════════════════════════════════════════╝

That appears to be a nice thing to have on your local workstation but not in a docker image. I tried to see if I can deactivate this backup or delete it afterward. There was no helpful paragraph in the man pages and searching online always directs me to some cloud database backup procedures you can initiate using gcloud commands.

Does anyone have further knowledge about what is happening here? Maybe it is not the backup at all that is contributing to most of the overhead?

gcloud docker google-cloud-platform
3 Answers

First I want to thank @gso_gabriel for his suggestions. I ended up analyzing filesystem utilization before and after calling gcloud components install.

  1. Getting rid of the backup turned out to be easy. In the gcloud application folder there is a hidden, hidden .install/.backup folder which was not there in the beginning and took up around 270 MB after component installation.
  2. Installing components also left behind a bunch of __pycache__ folders which summed up to around 100 MB.

To reduce the overall image size, I updated the corresponding docker line to:

RUN gcloud components install \
        cloud-datastore-emulator \
        pubsub-emulator \
        beta \
        --quiet \
    && rm -rf $(find google-cloud-sdk/ -regex ".*/__pycache__") \
    && rm -rf google-cloud-sdk/.install/.backup

The improvement can be seen with docker history:

CREATED BY                                      SIZE
/bin/sh -c gcloud components install        …   317MB

A reduction of over 300 MB. I tested the image by using it as datastore and pubsub service for one of my applications, running its unit-tests. Everything still appears to be working well.

There might be more files that can be cleaned but I believe I got the heavy-weights already.

user_32139810

First, I would recommend you to run the command gcloud components list, to get a full list of components that are available and currently installed in your SDK - this way, you should be able to check which one is getting all this size on your installation.

Besides that, as mentioned in the documentation Installing the Cloud SDK Docker image:

The Cloud SDK Docker Image is essentially Cloud SDK installed on top of a Debian-based OS image.

This means that probably part of the size is related to the OS of used for the installation.

I would like to also let you know that, it's not possible to not create the backup during the creation of the docker image.

For this reason, it might be worth it to contact Google directly, by raising a Feature Request, for this to be checked in the future - which would be good to have a better control over the components, such as the backup, for the image not be so big.

Let me know if the information helped you!

user_127672570

A bit of a late answer but what I ended up doing was modifying the google/cloud-sdk:alpine Dockerfile and added the code from the debian_component_based Dockerfile to install the components that I wanted as well as your fixes. This resulted in a docker image that only has the components that I needed while being smaller than the gcloud/cloud-sdk:alpine image.

Example:

RUN apk --no-cache add \
        curl \
        python3 \
        py3-crcmod \
        bash \
        libc6-compat \
        openssh-client \
        git \
        gnupg \
    && curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-${CLOUD_SDK_VERSION}-linux-x86_64.tar.gz && \
    tar xzf google-cloud-sdk-${CLOUD_SDK_VERSION}-linux-x86_64.tar.gz && \
    rm google-cloud-sdk-${CLOUD_SDK_VERSION}-linux-x86_64.tar.gz && \
    /google-cloud-sdk/install.sh --bash-completion=false --path-update=true --usage-reporting=false \
    --additional-components alpha && rm -rf $(find google-cloud-sdk/ -regex ".*/__pycache__") \
    && rm -rf google-cloud-sdk/.install/.backup

The resulting image size is 424 MB

user_29222950
Related