How to prevent docker build from redownloading copied Go vendor

user_1620210 0
Viewed 2655

I'm trying to make Dockerfile runs faster by copying whole directory (including vendor, because redownloading dependencies took about 10m+ in 3rd world country where I live), but when I tried to run it, it always redownload vendor again and again, unlike when go mod vendor in local:

FROM golang:1.14-alpine AS builder
RUN apk --update add ca-certificates git make g++
ENV GO111MODULE=on

WORKDIR /app

RUN go get github.com/go-delve/delve/cmd/dlv

COPY . .

RUN go mod vendor

ARG COMMIT_HASH
ENV COMMIT_HASH=${COMMIT_HASH}
ARG BUILD_DATE
ENV BUILD_DATE=${BUILD_DATE}

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
    go build \
    -o app

FROM golang:1.14-alpine

WORKDIR /app

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=builder /go/bin/dlv /
COPY --from=builder /app/app .
COPY --from=builder /app/db ./db
EXPOSE 8080 63342
CMD [ "/dlv", "--listen=:63342", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./app" ]

previously using this (without vendor) also slow:

COPY go.mod .
COPY go.sum .
RUN go mod download -x
COPY . .

trying with this also didn't work:

COPY vendor /go/pkg/mod
COPY vendor /go/pkg/mod/cache/download
COPY go.mod .
COPY go.sum .

RUN go mod download -x

COPY . .

how to force it to use copied vendor directory instead of redownloading again and again?

so the expected behavior are:

  1. when local have vendor (used go mod vendor), the docker build should use it
  2. but when on CI (since vendor/*/* not committed to the repo) or developer that doesn't have vendor/*/* it should probably redownload everything (I don't really care, since they have good bandwidth)

the go mod vendor command is for the CI and devs that haven't use go mod vendor

docker go caching
1 Answers

go mod vendor only download dependency from network if the dependency not ready in local. Otherwise, it will just copy dependency to the folder vendor without access network. So here, your issue comes from the go mod cache not be reused during multiple build.

As a solution, you could use buildkit cache solution, next is a minimal example:

main.go:

package main

import _ "github.com/jeanphorn/log4go"

func main() {
}

Dockerfile:

# syntax = docker/dockerfile:1.3
FROM golang:1.14-alpine AS builder

RUN apk --update add git

ENV GO111MODULE=on
WORKDIR /app

COPY main.go /app

RUN go mod init hello

RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go get github.com/go-delve/delve/cmd/dlv && go get github.com/jeanphorn/log4go
RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go mod vendor

1st Execution:

$ export DOCKER_BUILDKIT=1
$ docker build --progress=plain -t abc:1 . --no-cache
#16 [builder 6/7] RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go get github.com/go-delve/delve/cmd/dlv && go get github.com/jeanphorn/log4go
#16 sha256:ae394bc67787799808175eada48c5f4e09101b6e153d535ddb5e4040fbf74395
#16 1.941 go: downloading github.com/go-delve/delve v1.7.1
#16 4.296 go: found github.com/go-delve/delve/cmd/dlv in github.com/go-delve/delve v1.7.1
......
#16 23.78 go: finding module for package github.com/toolkits/file
#16 23.96 go: downloading github.com/toolkits/file v0.0.0-20160325033739-a5b3c5147e07
#16 24.17 go: found github.com/toolkits/file in github.com/toolkits/file v0.0.0-20160325033739-a5b3c5147e07
#16 DONE 27.3s

2nd Execution:

$ export DOCKER_BUILDKIT=1
$ docker build --progress=plain -t abc:1 . --no-cache
#15 [builder 6/7] RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go get github.com/go-delve/delve/cmd/dlv && go get github.com/jeanphorn/log4go
#15 sha256:bee74f92ceb79cce449b9702c892cb39815461981838f6b63d500414be87c21d
#15 1.467 go: found github.com/go-delve/delve/cmd/dlv in github.com/go-delve/delve v1.7.1
#15 7.511 go: github.com/jeanphorn/log4go upgrade => v0.0.0-20190526082429-7dbb8deb9468
#15 7.533 go: finding module for package github.com/toolkits/file
#15 7.675 go: found github.com/toolkits/file in github.com/toolkits/file v0.0.0-20160325033739-a5b3c5147e07
#15 DONE 8.7s

You could see golang mod cache generated by 1st run already be reused by 2nd run without downloading from internet, now it effects as same when you do it on host.

NOTE: I didn't suggest to directly bind any cache on host to container, it's not portable I think.

user_63947220
Related