Share same steps for different GitHub Actions jobs

Viewed 4452

I have a cross-platform project which is to be built on 2 platforms: mac and linux(ubuntu).

My pipeline contains 3 jobs:

  1. prepare docker image with all nessesary too to build the project.
  2. build on ubuntu in prepared docker container, depends on step 1
  3. build on MacOS, needs nothing

Steps for linux and macos are definitely the same. But matrixes differs much, and linux build is run inside container.

Is there a way to share steps between two different jobs?

I tried YAML anchors but GitHub does not support them.

Full workflow

on:
  push:
    branches: [ main, support/1.2.x ]
  pull_request:
    branches: [ main, support/1.2.x ]

jobs:
  Docker-iroha-builder:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Cache Docker layers
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-
      -
        name: Login to DockerHub
        uses: docker/login-action@v1 
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Build and push
        uses: docker/build-push-action@v2
        with:
          file: docker/develop/Dockerfile.builder
          # context: .
          push: true
          tags: ${{ secrets.DOCKERHUB_ORG }}/iroha:builder
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new
      -
        # Temp fix
        # https://github.com/docker/build-push-action/issues/252
        # https://github.com/moby/buildkit/issues/1896
        name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache


  build-iroha-ubuntu:
    needs: Docker-iroha-builder
    runs-on: ubuntu-latest
    container: ikyb/iroha:builder
    strategy:
      fail-fast: false
      matrix:
        cc: [ gcc-9, gcc-10, clang ]  ##todo g++-10
        USE_BURROW: [ -DUSE_BURROW=OFF ]
        debrel: [ Debug ] #,Release, RelWithDebInfo
    steps:
      - ## Takes 22 seconds with default github runner
        name: Homebrew
        run: brew install cmake ninja coreutils
        if: ${{ runner.os == 'MacOS' }}
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Cache vcpkg
        uses: actions/cache@v2
        with:
          path: |
            build-vcpkg
            build/vcpkg_installed
            $HOME/.cache/vcpkg
          key:          ${{ runner.os }}-vcpkg-${{ github.sha }}
          restore-keys: ${{ runner.os }}-vcpkg-
      - 
        name: Build Iroha vcpkg dependancies
        run: ./vcpkg/build_iroha_deps.sh $PWD/build-vcpkg
      - 
        name: CMake configure
        run: |
          export CC=${{ matrix.cc }} CXX=$(echo ${{ matrix.cc }} | sed -es,gcc,g++, -es,clang,clang++,)
          cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/build-vcpkg/scripts/buildsystems/vcpkg.cmake \
             ${{ matrix.USE_BURROW }} -GNinja #-DCMAKE_VERBOSE_MAKEFILE=ON
      -
        name: CMake build
        run: cmake --build build --config ${{ matrix.debrel }}

  build-iroha-macos:
    runs-on: macos-latest
    strategy:
      fail-fast: false
      matrix:
        USE_BURROW: [ -DUSE_BURROW=OFF ]
        debrel: [ Debug,Release ]
    steps:
      - ## Takes 22 seconds with default github runner
        name: Homebrew
        run: brew install cmake ninja coreutils
        if: ${{ runner.os == 'MacOS' }}
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Cache vcpkg
        uses: actions/cache@v2
        with:
          path: |
            build-vcpkg
            build/vcpkg_installed
            $HOME/.cache/vcpkg
          key:          ${{ runner.os }}-vcpkg-${{ github.sha }}
          restore-keys: ${{ runner.os }}-vcpkg-
      - 
        name: Build Iroha vcpkg dependancies
        run: ./vcpkg/build_iroha_deps.sh $PWD/build-vcpkg
      - 
        name: CMake configure
        run: |
          export CC=${{ matrix.cc }} CXX=$(echo ${{ matrix.cc }} | sed -es,gcc,g++, -es,clang,clang++,)
          cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/build-vcpkg/scripts/buildsystems/vcpkg.cmake \
             ${{ matrix.USE_BURROW }} -GNinja #-DCMAKE_VERBOSE_MAKEFILE=ON
      -
        name: CMake build
        run: cmake --build build --config ${{ matrix.debrel }}

1 Answers

TL;DR

I solved my problem with shell tool yq

yq eval 'explode(.)' file.yml

The repository with example usage and detailed description https://github.com/kuvaldini/make-workflows.sh may be helpful to easy start. It was make from this answer. Pay attention to Actions tab.

Long answer

GitHub Workflow description in YAML does not support anchors. There are several workarounds => anyway they come to building-editing workflow yaml from source. So I suggest yet another one make-workflows.sh based on YAML tool yq.

USAGE

  1. Move your workflows to .github/*.src.yml
  2. Put make-workflows.sh to directory .github/
  3. (optional) Copy or link pre-commit.sh to .git/hooks/pre-commit Like ln -s ../../.github/pre-commit.sh .git/hooks/pre-commit

File make-workflows.sh

#!/usr/bin/env bash
set -euo pipefail

## The script expands '*.src.yml' from $1(default: script's directory) 
## to $2 (default:subdirectory 'workflows') with corresponding name '*.yml'
## Main goal is to dereference YAML anchors.
## Deals only with Git cached/indexed files
## Set -x to debug

script_dir=$(dirname $(realpath "$0"))
dir_from=${1:-${script_dir}}
dir_to=${2:-workflows}
cd $dir_from

edited=
for f in $(git status -s -- \*.src.yml | sed 's,^.. ,,') ;do
    readonly out=$(echo $f | sed s,.src.yml\$,.yml,)
    readonly wout=$dir_to/$out
    readonly tempout=$(mktemp)
    trap "rm -f $tempout" EXIT
    echo >>$tempout "## DO NOT EDIT"
    echo >>$tempout "## Generated from $f with $(basename $0)"
    echo >>$tempout ""
    yq eval 'explode(.)' $f >>$tempout
    if ! diff -q $wout $tempout &>/dev/null ;then
        mv $tempout $wout
        edited+="'$out' "
    fi
done

if [[ -n "$edited" ]] 
then echo >&2 "make-workflows: these files were edited: $edited"
else echo >&2 "make-workflows: everything is up to date"
fi

File pre-commit.sh

#!/usr/bin/env bash
set -euo pipefail

gitroot=$(git rev-parse --show-toplevel)

cd $gitroot
./.github/make-workflows.sh
git add .github/workflows

Links

  1. ready to use solution with detailed description https://github.com/kuvaldini/make-workflows.sh
  2. Share same steps for different GitHub Actions jobs
  3. https://github.community/t/support-for-yaml-anchors/16128/60
  4. https://github.com/mithro/actions-includes
  5. https://github.com/allejo/gha-workflows
Related