GitLab CI Yocto Build - How to use SSTATE and DL_DIR

user_5415668 0
Viewed 3131

How to configure GitLab CI to store the SSTATE_DIR and the DL_DIR between several jobs? Currently, bitbake rebuilds the complete project every time, which is very time consuming. So i would like to use the sstage again. I tried caching, but building time increases effectively, due to the big zip/unzip overhead.

I don't even need a shared sstate between several projects, just a method to store the output between jobs.

I'm using Gitlab 11.2.3 with a shell executor as runner.

Thanks a lot!

bitbake yocto gitlab gitlab-ci-runner
3 Answers

If you're only using one runner for this, you could potentially use GIT_STRATEGY: none, which would re-use the project workspace for the following job; relevant documentation. However, this wouldn't be extremely accurate in case you have multiple jobs running which requires the runner, as it could dilute the repository, if jobs are started from across different pipelines.

Another way, if you're still using one runner, is you could just copy the directories out and back into the job you require.

Otherwise, you may potentially be out of luck, and have to wait for the sticky runners issue.

user_40474720

You can reuse a shared-state cache between jobs simply as follows:

Specify the path to the sstate-cache directory in the .yml file of your gitlab-ci pipeline. An example fragment from one of mine:

myrepo.yml

stages:
  ...
  ...

variables:
  ...
  TCM_MACHINE: buzby2
  ...
  SSTATE_CACHE: /sstate-cache/$TCM_MACHINE/PLAT3/
  PURGE_SSTATE_CACHE: N
  ...

In my case /sstate-cache/$TCM_MACHINE/PLAT3/ is a directory in the docker container that runs the build. This path is mounted in the docker container from a permanent sstate cache directory on the build-server's filesystem, /var/bitbake/sstate-cache/<machine-id>/PLAT3.

PURGE_SSTATE_CACHE is overridable by a private variable in the pipeline schedule settings so that I can optionally delete the cache for a squeaky clean build.

Ensure that the setting of SSTATE_CACHE is appended to the bitbake conf/local.conf file of the build, e.g.

.build_image: &build_image
  stage: build
  tags:
    ...
  before_script:
    ...
  script:
    - echo "SSTATE_DIR ?= \"$SSTATE_CACHE\"" >> conf/local.conf
    ...

Apply the same pattern to DL_DIR if you use it.

Variables you use in the .yml file can be overriden by gitlab-ci trigger or schedule variables. See Priority of variables

user_13625680

In version 11.10, GIT_CLEAN_FLAGS was added, which could be used to achieve what you want to do with the shell executor.

For completeness: when using the docker executor, this can be achieved by using docker volumes, which are persistent across builds.

user_17103920
Related