Gitlab CI: run job only if artifact exists

Question

I have monorepo and I want to run child pipeline depending on content of directory which has changed. In job prepare_config I check where are latest changes, I create child config yml and in next stage's job run_child I run child pipeline from .

The problem is, if model-gitlab-ci.yml doesn't exist, then job run_child fails instead of skipping due to missing artifact. I searched for solution to conditionally run job only if artifact exists instead of failing, but didn't found any solution. Maybe someone here have some idea?

.gitlab-ci.yml:

stages:
  - .pre
  - build

prepare_config:
  stage: .pre
  tags:
    - sometag
  rules:
    - if: $CI_COMMIT_TAG == null
      when: always
      changes:
      - '.gitlab-ci.yml'
      - 'DIR_A/**/*'
      - 'DIR_B/**/*'
      - 'DIR_C/**/*'
  script:
    - |-
      files=$(git diff-tree --name-only --no-commit-id ${CI_MERGE_REQUEST_TARGET_BRANCH_SHA:-$CI_COMMIT_SHA})
      echo "Files changed: $files"
      for f in $files; do
        if [ -d $f ]; then
          sed "s/{{ MODEL_NAME }}/$f/g" .gitlab-ci-template.yml >> model-gitlab-ci.yml
        fi
      done
  artifacts:
    paths:
      - "model-gitlab-ci.yml"

run_child:
  stage: build
  rules:
    - if: $CI_COMMIT_TAG == null
      when: always
  needs:
    - job: prepare_config
      artifacts: true
  trigger:
    include:
      - artifact: model-gitlab-ci.yml
        job: prepare_config
    strategy: depend

user_3535187 · Answer

The job "jsonnet" uses jsonnet to generate a YAML file "trigger.yml" from the file "trigger.jsonnet". If the variable in the JSONNET file triggering == true it contains the jobs "artifact:helper" and "trigger". If triggering == false it contains the job "not:triggered", which runs empty.

The job "artifact:helper" is required so that the generated file "generated.yml" can be transferred to the trigger as an artifact. If a trigger job in a child pipeline wanted to use an artifact of the parent pipeline directly, a cross-dependency exception would occur.

The job "trigger" in the YAML file "trigger.yml" ultimately triggers the job "template_job" from "generated.yml".

Jobs if triggering == true

Jobs if triggering == false

.gitlab-ci.yml

jsonnet:
  image: jkblskw/jsonnet:jsonnet-curl
  stage: .pre
  script:
    # do something
    - if [ "$TEST_TRIGGER" = true ]; then TRIGGERING=true && cp template.yml generated.yml; else TRIGGERING=false; fi
    - jsonnet --ext-code triggering=$TRIGGERING --ext-code pipeline_id=$CI_PIPELINE_ID trigger.jsonnet > trigger.yml
  artifacts:
    paths:
      - trigger.yml
      - generated.yml

trigger:
  stage: build 
  trigger:
    include:
      - artifact: trigger.yml
        job: jsonnet
    strategy: depend

trigger.jsonnet

local triggering = std.extVar("triggering");
local pipeline_id = std.extVar("pipeline_id");

if triggering then 
{
    ["artifact:helper"]: {
        stage: "build",
        variables: {
            PARENT_PIPELINE_ID: pipeline_id
        },
        needs:{
            pipeline: "$PARENT_PIPELINE_ID",
            job: "jsonnet"
        },
        script: "echo 'Job needs to be executed in order for the artifact to be triggered. A trigger in the same job throws a cross-reference exception.'",
        artifacts: {
            paths: ["generated.yml"]
        }
    },
    ["trigger"]: {
        stage: "test",
        needs: ["artifact:helper"],
        trigger: {
            include: {
                artifact: "generated.yml",
                job: "artifact:helper"
            },
            strategy: "depend"
        }
    } 
}else
{
    ["not:triggered"]: {
        stage: "build",
        script: "echo 'no further triggering'"
    } 
}

template.yml

template_job:
  stage: test
  script: 
      - echo "running template_job"

Gitlab CI: run job only if artifact exists

1 Answers