Jenkins declarative pipeline: Sequential execution of an axis of a matrix-build

user_11415117 0
Viewed 2564

I am trying to set up a matrix build and am running in several issues. The matrix looks like:

stage( 'BuildAll' ) {
    matrix {
        axes {
            axis {
                name 'PLATFORM'
                values 'win32vc9', 'win32vc19' 
            }                   
            axis {
                name 'VARIANT'
                values 'debug', 'release'                   
            }
            axis {
                name 'MODULES'
                values 'repo1/module1', 'repo1/module2', 'repo2/module1', 'repo2/module2'
            }
        }

        stages {                                                
            stage( 'Checkout' ) {
            }
            stage( 'Build' ) {
            }
            stage( 'Test' ) {
            }
       }

The issue I have:

  • jenkins is executing every cell of the matrix in it's own workspace but my modules depend on each other. That's why I want do check them out and build them in the same workspace. BTW: my build system is made to work with all the variants (debug/release x vc9/vc19) in the same workspace.

  • jenkins is executing all the cells in parallel. What I need, is a serialization of execution of the MODULES axis.

Any ideas how I can work around this?

E.g. is there a way to descripe something like a loop over several modules that generates a sequence of stages in a row, not parallel? Within that sequence I could realize the matrix over 2 axes only.

I am aware of extended workspace plugin but did not find any documentation of how to use this in declarative pipelines.

Many thanks in advance! Regards, Christoph

jenkins-pipeline declarative jenkins matrix
3 Answers

I used for loop instead of matrix and axis syntax. My workaround is not as pretty as using matrix syntax, but it does what it supposed to do.

String[] platformList = ['win32vc9', 'win32vc19']
String[] variantList = ['debug', 'release']

pipeline{
    agent any
    stages {
        stage('Deploy'){
            steps{
                script{
                    for(platform in platformList){
                        for(variant in variantList){
                            stage("Checkout"){
                                powershell label: '', script: """Write-Output Step1 "${platform}/${variant}/Checkout" """
                            }
                            stage("Build"){
                                powershell label: '', script: """Write-Output Step1 "${platform}/${variant}/Build" """
                            }
                            stage("Test"){
                                powershell label: '', script: """Write-Output Step1 "${platform}/${variant}/Test" """
                            }
                        }
                    }
                }
            }
        }
    }
}
user_95468500

You might be able to do what you want by using locks.

In my case, I just wanted the matrix to build sequentially instead of in parallel, so I just wrapped everything in a stage and put a lock on it. Something like:

stage( 'BuildAll' ) {
    matrix {
        axes {
            ...
        }
        stages {                                                
            stage( 'Sequential Matrix' ) {
                options {
                    lock( 'synchronous-matrix' )
                }
                stages {
                    ...
                }
            }
        }
    }
}

Only one matrix configuration can build at a time since the others can't acquire the lock until the current config is done.

In your case though, you want the modules to build one after the other in order. The string for the lock resource can be dynamic, so I think you could use a name constructed from the other 2 axes to prevent multiple modules from building at the same time:

stage( 'BuildAll' ) {
    matrix {
        axes {
            axis {
                name 'PLATFORM'
                values 'win32vc9', 'win32vc19' 
            }                   
            axis {
                name 'VARIANT'
                values 'debug', 'release'                   
            }
            axis {
                name 'MODULES'
                values 'repo1/module1', 'repo1/module2', 'repo2/module1', 'repo2/module2'
            }
        }

        stages {                                                
            stage( 'Force Sequential' ) {
                options {
                    lock( "synchronous-${PLATFORM}-${VARIANT}" )
                }
                stages {
                    stage( 'Checkout' ) {

                    }
                    stage( 'Build' ) {

                    }
                    stage( 'Test' ) {

                    }
                }
            }
        }
    }
}

Note that this would only guarantee modules of the same platform and variant don't build at the same time and doesn't say anything about ordering. I don't know if the order which Jenkins chooses to build axes is defined, but I found matrix-sorter-plugin which may help with ordering if it's an issue.

user_16756680

Declarative Pipeline matrix directive doesn't currently support sequential execution.
The best you can do in in the current matrix syntax is this: 

 stage( 'BuildAll' ) {
    matrix {
        axes {
            axis {
                name 'PLATFORM'
                values 'win32vc9', 'win32vc19' 
            }                   
            axis {
                name 'VARIANT'
                values 'debug', 'release'                   
            }
        }
        stages {                
            stage('repo1/module1') {
                stages {
                    stage( 'Checkout' ) {
                    }
                    stage( 'Build' ) {
                    }
                    stage( 'Test' ) {
                    }
                }
            }
            stage('repo1/module2') {
                stages {
                    stage( 'Checkout' ) {
                    }
                    stage( 'Build' ) {
                    }
                    stage( 'Test' ) {
                    }
                }
            }
            stage('repo2/module1') {
                stages {
                    stage( 'Checkout' ) {
                    }
                    stage( 'Build' ) {
                    }
                    stage( 'Test' ) {
                    }
                }
            }
            stage('repo2/module2') {
                stages {
                    stage( 'Checkout' ) {
                    }
                    stage( 'Build' ) {
                    }
                    stage( 'Test' ) {
                    }
                }
            }
        }
    }
}

Others have asked for similar. I've filed JENKINS-62085 based on this question.

user_16372520
Related