Building a rust project with docker is extremely slow on google cloud

Question

I'm relatively new to Rust but I've been working a project within a Docker container. Below is my dockerfile and it works great. My build uses an intermediary container to build all the cargo containers before the main project. Unless I update a dependency the project builds very quickly locally. Even with the dependencies getting rebuilt it doesn't take more than 10 minutes max on my old macbook pro.

FROM ekidd/rust-musl-builder as builder

WORKDIR /home/rust/

# Avoid having to install/build all dependencies by copying
# the Cargo files and making a dummy src/main.rs
COPY Cargo.toml .
COPY Cargo.lock .
RUN echo "fn main() {}" > src/main.rs
RUN cargo test
RUN cargo build --release

# We need to touch our real main.rs file or else docker will use
# the cached one.
COPY . .
RUN sudo touch src/main.rs

RUN cargo test
RUN cargo build --release

# Size optimization
RUN strip target/x86_64-unknown-linux-musl/release/project-name

# Start building the final image
FROM scratch
WORKDIR /home/rust/
COPY --from=builder /home/rust/target/x86_64-unknown-linux-musl/release/project-name .
ENTRYPOINT ["./project-name"]

However, when I set up my project to automatically build from the github repo via google cloud build I was shocked to see builds taking almost 45 minutes! I figured if I got the caching setup properly for the intermediary container at least that would shave some time off. Even though the builder successful pulls the cached image it doesn't seem to use it and always build the intermediary container from scratch. Here is my cloudbuild.yaml:

steps:
  - name: gcr.io/cloud-builders/docker
    args:
      - "-c"
      - >-
        docker pull $_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:latest
        || exit 0
    id: Pull
    entrypoint: bash
  - name: gcr.io/cloud-builders/docker
    args:
      - build
      - "-t"
      - "$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:latest"
      - "--cache-from"
      - "$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:latest"
      - .
      - "-f"
      - Dockerfile
    id: Build
  - name: gcr.io/cloud-builders/docker
    args:
      - push
      - "$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:latest"
    id: Push
  - name: gcr.io/google.com/cloudsdktool/cloud-sdk
    args:
      - run
      - services
      - update
      - $_SERVICE_NAME
      - "--platform=managed"
      - "--image=$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:latest"
      - >-
        --labels=managed-by=gcp-cloud-build-deploy-cloud-run,commit-sha=$COMMIT_SHA,gcb-build-id=$BUILD_ID,gcb-trigger-id=$_TRIGGER_ID,$_LABELS
      - "--region=$_DEPLOY_REGION"
      - "--quiet"
    id: Deploy
    entrypoint: gcloud
timeout: 3600s
images:
  - "$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:latest"
options:
  substitutionOption: ALLOW_LOOSE

I'm looking for any info about what I'm doing wrong in my cloudbuild.yaml and tips on how to speed up my cloud builds considering it's so fast locally. Ideally I'd like to stick with google cloud but if there is another CI service that handles rust/docker builds like this better I'd be open to switch.

user_567987 · Answer

This is what I did to improve build time Rust projects on Google Cloud Build. Not a perfect solution, but better than nothing:

Similar changes to yours in Docker file to create different cache layers for deps and my own sources.
Used kaniko to leverage caching (this seems your particular issue)

  steps:
 - name: 'gcr.io/kaniko-project/executor:latest'
    args:
    - --destination=eu.gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA
    - --cache=true
    - --cache-ttl=96h
    timeout: 2400s

Docs: https://cloud.google.com/build/docs/kaniko-cache

Changed machine type to higher options, in my case:

options:
  machineType: 'E2_HIGHCPU_8'

Be careful though, changing machine types will affect your budgets, so you should consider if this worth it for your particular project.

If you push frequently your changes this works much better, yet still not good enough to be honest.

user_11372593 · Answer

There is 2 things to consider in term of speed:

On your (even old) macbook pro,
- You have multi core hyperthreaded CPU
- The CPU can go up to 3.5Ghz in turbo mode
On Cloud Build
- You have only one vCPU per build (by default)
- The vCPU are "server designed CPU": no highend performance, but stable and consistent performance, around 2.1Ghz (slightly more in turbo mode)

So, the difference of performance is obvious. To speed up your build I can recommend to use the machine type option:

...
...
options:
  substitutionOption: ALLOW_LOOSE
  machineType: 'E2_HIGHCPU_8'

It should be better!

Building a rust project with docker is extremely slow on google cloud

2 Answers