Gitlab CI in multiple platforms simultaneously

user_3775425 0
Viewed 8146

I have a C++ project that is compiled and packaged for multiple OS (Linux, Windows, MacOS) as well as multiple CPU architectures (i386, x86_64, arm, Aarch64) For this I'm using Jenkins to grab the source code and run the build script in parallel on each system. It's a simple working solution, since my build script deals with the system differences.

Now I'm looking into Gitlab CI/CD, and it has many things I find appealing ( being able to keep the build script as part of the repository, very well integrated with the git repo and ticketing system, natural use of Docker containers, etc), but I cannot find any way to run the same pipeline in multiple architectures/systems parallel to each other.

So, say that my build script is:

build:
  stage: build
  script: 
    - uname -m > arch.txt
  artifacts:
    paths:
      - arch.txt

How can I tell Gitlab that I want to run this job in multiple runners/Docker containers/systems at the same time? All the documentation I've read so far deals with running multiple tests on one build, integrating multiple projects or deploying in different environments depending on branches. Nothing I've read so far tries to do many separate builds, test and package them individually and report on their independent results. Is this feasible on Gitlab CI/CD?

gitlab gitlab-ci pipeline
2 Answers

GitLab uses "runners" to execute CI jobs. Runners are installed wherever you want to run a CI job, so if you want to run on multiple architectures then you will need to install runners on systems for each architecture. Runner install documentation can be found here:

https://docs.gitlab.com/runner/install/index.html

For Linux-based jobs it is common to use Docker for job execution - this doesn't give architectural flexibility, but it does allow you to test on different flavors and with different software using containerisation. For other architectures you may need to install runners yourself, or use other peoples shared runners.

While you are installing the runner software there are some keys steps:

  • you have the opportunity to link each runner to your GitLab project, which means it will show up in the runners list under Project > Settings > CI/CD.

  • you will have the opportunity to assign "tags" to the runners. Tags can be used to help identify a runner or group of runners by an arbitrary name (e.g. you could add "Windows x86_64" as a tag, or "Windows" and "x86_64" tags). These tags can be used in jobs to select a runner.

Once you have your runners installed you can get editing your .gitlab-ci.yml file.

GitLab CI files are broken up into "stages". Jobs in each stage can run in parallel. Stage names are defined at the top of the file.

stages:
  - build
  - deploy

Each CI job can be attached to a stage by using the stage: entry:

build job:
  stage: build
  script:
    - echo "I am a build stage job"

In your case you will need to create multiple jobs for each architecture you want to build for. Attaching these to the same stage will allow them to run in parallel.

To control where each job runs you have two main mechanisms:

  1. Tags - tags allow you to pin a job to a runner tag. You can specify multiple tags using the tags: entry which forms an AND list (e.g. win tag AND x86_64 tag). When that job runs GitLab will find a runner that has all the required tags, and run the job there.

  2. Image - When running on Docker / Kubernetes you can specify a docker image to use for the runner. To use a docker image you first need to specify a runner that can run docker images (e.g. a docker-in-docker or kubernetes runner), which might, for example, be tagged with docker or kubernetes. Then you use the image: entry to specify the docker image.

Here's an example showing both tags and images:

build win x86_64:
  stage: build
  tags:
    - win
    - x86_64
  script:
    - echo "I am a build stage job for win x86_64"

build win 32:
  stage: build
  tags:
    - win
    - 32-bit
  script:
    - echo "I am a build stage job for win 32"

build debian:
  stage: build
  tags:
    - docker
  image: debian:stretch
  script:
    - echo "I am a build stage job for debian, running on docker using debian:stretch image"

There is currently no support for dynamic jobs, or running one job on multiple runners / architectures, so this involves a bit of manual effort. On the positive side it makes GitLab CI files easy to read, and easy to see what will run during CI execution.

user_62525250

Check out the latest GitLab release 11.5 (Nov. 2018) with:

Parallel attribute for faster pipelines

The speed of pipelines is an important factor for any team, and running tests or other parallelizable tasks tends to take a big chunk of the time for any build.

Adding this new keyword gives teams the ability to simply parallelize tests, allowing everyone to accelerate their software delivery process.
To use this feature, simply set the parallel attribute to how many copies of the task you’d like to split it into, and GitLab will handle the work of automatically creating the appropriate number of jobs running your task.

As commented by March, this is not an exact fit for the OP (which is about running the same pipeline in multiple architectures/systems parallel to each other).

See documentation and issue.

parallel allows you to configure how many instances of a job to run in parallel.
This value has to be greater than or equal to two (2) and less or equal than 50.

This creates N instances of the same job that run in parallel.
They’re named sequentially from job_name 1/N to job_name N/N.

test:
  script: rspec
  parallel: 5
VonC_63090
Related