How to make gitlab run jobs in sequential order?

user_1581090 0
Viewed 2476

I have a gitlab-ci.yml file like the following:

stages:
  - test
  - job1
  - job2


test:
  stage: test
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  script:
    ...

myjob1:
  stage: job1
  script:
    ...


myjob2:
  stage: job2
  script:
    ...

According to the documentation HERE (or at least how I understood it), the first stage/job is only run when I create a merge request.

This is true, but the next stage (job1) is run in parallel when the first job (test) has been started. As far as I understand the stages which are defined in the order test -> job1 -> job2) always run in sequence.

So what am I doing wrong? Why do job test and job1 run in parallel, and not in sequence as expected?

gitlab
2 Answers

After a lot of trial-end-errors and reading and rereading parts of the really unclear and confusing documentation I might have found a solution.

First, the stage you only want to run on a merge request (or which you do not want to run if you schedule a trigger or start the pipeline manually), you need to change that stage as follows:

test:
  rules:
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "schedule"
      when: never
    - when: on_success
  stage: test
  script:
    - 'echo "Running Test"'
    - 'echo $CI_PIPELINE_SOURCE'

Here, you define a rule that checks if the variable CI_PIPELINE_SOURCE is either web or schedule. If the variable is web this means a manual pipeline trigger (i.e. you pressed manually on Run pipeline, which is not explained in the documentation), or if the pipeline is triggered by a schedule (not tested, but I assume that is what schedule means).

So if the pipeline is triggered by a scheduled event or manually, never tells gitlab to not execute that stage. The when: on_success is like an else statement, which tells gitlab to run that stage in any other case.

However, that is not the complete story. Because when you use git to make changes to the code and push it to gitlab via git push, you have two triggers in gitlab! A trigger merge_request_event and a trigger push. That means, the pipeline is started twice!

To avoid the pipeline started twice you need to use the workflow key, which helps to define if the pipeline (=workflow) is run or not. (The term workflow seems to mean pipeline). Here is the code to be put into the gitlab-ci.yml file:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
    - when: always

This construct suppresses the pipeline to be run when the trigger is merge_request_event. In that case, the additional pipeline is not run. In all other cases (when the trigger is e.g. push), the pipeline is run.

So here is the complete gitlab-ci.yaml code:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
    - when: always


stages:
  - test
  - stage1
  - stage2


test:
  rules:
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "schedule"
      when: never
    - when: on_success
  stage: test
  script:
    - 'echo "Running Test"'

my_stage1:
  stage: stage1
  script:
    - 'echo "Running stage 1"'


my_stage2:
  stage: stage2
  script:
    - 'echo "Running stage 2"'

If you make a git push then one pipeline is run with the stages test, my_stage1 and my_stage2, and when you start the pipeline manually or if it is triggered by a schedule, one pipeline is started with the stages my_stage1 and my_stage2.

As to why this is so complicated and confusing, I have not the slightest idea.

user_15810900

The 2016 blog post "GitLab CI: Run jobs sequentially, in parallel or build a custom pipeline" from Ivan Nemytchenko suggests using stages indeed.
That was revised in Dec. 2020 in "The basics of CI: How to run jobs sequentially, in parallel, or out of order" by Itzik Gan Baruch.

Maybe there is more than one runner (because of duplicate pipelines), giving the illusion of different stages of the pipeline running. In which case, you can add a tag to your stages, with concurrency set to 1.

The idea, expressed in "Avoid duplicate pipeline", is to avoid the same pipeline is run twice.

In your case, as detailed in "Switch between branch pipelines and merge request pipelines" is to:

  • make sure your pipeline does not run when there us a change to a branch, but a merge request is open for that branch ("branch pipeline" = "run where there is a change to the branch": to be avoided in your case)
  • make sure your pipeline does run when there is a merge request event ("merge request pipeline")

That is what the following workflow does:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
      when: never

That would help making sure the pipeline is not triggered twice, avoiding any "job1" stage (seemingly) running in paralel of test.

VonC_63090
Related