Groking the parallel-matrix

user_2245633 0
Viewed 5096

This seems to be a mis-use of the new Parallel Matrix feature in GitLab 13.3 (https://docs.gitlab.com/ee/ci/yaml/#parallel-matrix-jobs)

I have a collection of parallel jobs for a set of services: build (docker image), test, release, delete.... and the code-base is created so that each parallel service is in a separate sub-directory.

This way I can have a common template:

variables:
  IMAGE_NAME: $CI_REGISTRY_IMAGE/$LOCATION

.build-template:
  script:
    - docker build --tag $IMAGE_NAME:$CI_PIPELINE_ID-$CI_COMMIT_REF_SLUG --tag $IMAGE_NAME:latest $LOCATION
  stage: build
  when: manual

then multiple build jobs:

build-alpha:
  extends: .build-template
  variables:
    LOCATION: alpha

build-beta:
  extends: .build-template
  variables:
    LOCATION: beta

.... and repeat as needed.

I can then do the same thing for test, release, and delete jobs: a common template that takes just the one variable to distinguish the service.

Matrix to the rescue?

It would seem to me that

variables:
  IMAGE_NAME: $CI_REGISTRY_IMAGE/$NOTEBOOK

build-services:
  parallel:
    matrix:
      - LOCATION: alpha
      - LOCATION: beta
  script:
    - docker build --tag $IMAGE_NAME:$CI_PIPELINE_ID-$CI_COMMIT_REF_SLUG --tag $IMAGE_NAME:latest $LOCATION
  stage: build
  when: manual

would be an ideal candidate for this matrix form.... but apparently matrix requires two variables.

Has anyone got a good solution for this multiple-parallel jobs problem?

gitlab-ci
3 Answers

would be an ideal candidate for this matrix form.... but apparently matrix requires two variables.

Not anymore.

See GitLab 13.5 (October 2020)

Allow one dimensional parallel matrices

Previously, the parallel: matrix keyword, which runs a matrix of jobs in parallel, only accepted two-dimensional matrix arrays. This was limiting if you wanted to specify your own array of values for certain jobs.

In this release, you now have more flexibility to run your jobs the way that works best for your development workflow.
You can run a parallel matrix of jobs in a one-dimensional array, making your pipeline configuration much simpler. Thanks Turo Soisenniemi for your amazing contribution!

Here’s a basic example of this in practice that will run 3 test jobs for different versions of Node.js, but you can apply this approach to your specific use cases and easily add or remove jobs in your pipeline as well:

https://about.gitlab.com/images/13_5/simple-parallel-matrices-example.png -- One dimensional matrices example

See Documentation and Issue.

And with GitLab 13.10 (March 2021)

Use 'parallel: matrix' with trigger jobs

You can use the parallel: matrix keywords to run jobs multiple times in parallel, using different variable values for each instance of the job.

Unfortunately, you could not use it with trigger jobs.

In this release, we’ve expanded the parallel matrix feature to support trigger jobs as well, so you can now run multiple downstream pipelines (child or multi-project pipelines) in parallel using different variables value for each downstream pipeline.
This lets you configure CI/CD pipelines that are faster and more flexible.

https://about.gitlab.com/images/13_10/parallel.png -- Use 'parallel: matrix' with trigger jobs

See Documentation and Issue.

VonC_63090

Did you try:

parallel:
  matrix:
    - LOCATION: [alpha, beta]
user_25092800

This is a known issue by Gitlab.

There is a workaround using a "dummy" value for the second value (I'm working with Gitlab 13.3.1).

For your example, it gives this :

parallel:
  matrix:
    - LOCATION: ['alpha', 'beta']
      DUMMY: 'dummy'
user_26539110
Related