Bitbucket docker image cache for two parallel docker builds

user_14625928 0
Viewed 1104

I was tinkering with bitbucket pipelines for some time and have come far in improving time and resources taken for a buid/deployment script. But i got stuck at this Problem:

  - parallel:
    - step:
        caches:
          - docker
        services:
          - docker
        script:
          - echo ${REGISTRY_PASSWORD} | docker login registry.our-domain.net --username bitbucket --password-stdin &&
          - docker build --rm -t registry.our-domain.net/project-container_2:staging -f docker/web/Dockerfile . &&
          - docker push registry.our-domain.net/project-container_2:staging
    - step:
        caches:
          - docker
        services:
          - docker
        script:
          - echo ${REGISTRY_PASSWORD} | docker login registry.our-domain.net --username bitbucket --password-stdin &&
          - docker build --rm -t registry.our-domain.net/project-container_1:staging -f docker/app/Dockerfile . &&
          - docker push registry.our-domain.net/project-container_1:staging

If i try to cache both image build processes, i cant. Only the build process that is compiled first will be cached.

Skipping assembly of docker cache as one is already present Cache "docker": Skipping upload for existing cache

It seems like the caching is done based on the name of the defined cache. the docker-cache is done by bitbucket, so i cant "rename" the second one.

Does somebody have the same issue/ a solution ?

docker bitbucket caching
1 Answers

I had similar issue with Docker cache in BitBucket pipelines. Once the cache is built and uploaded, it won't get updated unless it is deleted. During parallel builds there is a race condition so the quickest step wins.

Fortunatelly, there is a way to manage cache the other way, but it requires some more scripting. Just make use of Docker's save and load commands and use custom cache locations defined in caches section. Thanks to that, each step is going to have its own cache.

I have embedded the same script in two places just for the sake of clarity, but you would probably like to use some YAML anchors and parametrize it a bit, in order to make this solution more generic.

Additionally, if cache gets too big you might just save some of the images being produced by the Docker, instead of saving all of them (docker save imageName instead of docker save $(docker images -qa))

definitions:
  caches:
    custom-docker-cache-one: custom-docker-cache-one
    custom-docker-cache-two: custom-docker-cache-two
  steps:
    - step-one:
        services:
          - docker
        caches:
          - custom-docker-cache-one
        script: |-
          if [ -f "custom-docker-cache-one/cache.tar" ]; then
            docker load < custom-docker-cache-one/cache.tar;
          fi

          # Do some operations here

          if [ ! -f "custom-docker-cache-one/cache.tar" ]; then
            docker save $(docker images -qa) > custom-docker-cache-one/cache.tar;
          fi
    - step-two:
        services:
          - docker
        caches:
          - custom-docker-cache-two
        script: |-
          if [ -f "custom-docker-cache-two/cache.tar" ]; then
            docker load < custom-docker-cache-two/cache.tar;
          fi

          # Do some operations here

          if [ ! -f "custom-docker-cache-two/cache.tar" ]; then
            docker save $(docker images -qa) > custom-docker-cache-two/cache.tar;
          fi
user_150449720
Related