Jenkins 2 dimensional array parallel stages

MatthewKim 0
Viewed 60

I am trying to implement Jenkins 2D parallel stages. This is the example that I am testing. I expect 12 different stages which are A1/B1/C1/A2/B3/C2/A3/B3/C3/A4/B4/C4 and they are executing 3 parallel builds and each build has 4 serial build stages.

    build_list = ['B1', 'B2', 'B3']

    l1 = ['A1', 'A2', 'A3', 'A4']
    l2 = ['B1', 'B2', 'B3', 'B4']
    l3 = ['C1', 'C2', 'C3', 'C4']

    array_build_list = [ l1, l2, l3 ]

    index = 1
    index = -index

    def parallelStagesMap = build_list.collectEntries {
        index++
        ["${it}" : generateStage(it, index)]
    }

    def generateStage(job, index) {
        return {
            stage("${job}") {
                echo "This is ${job}"
                script {
                    build_project_list = array_build_list[index]
                    echo "build_project_list = ${build_project_list}"
                    
                    for(int i=0; i<build_project_list.size(); i++) {
                        stage(build_project_list[i]) {
                            echo "${build_project_list[i]}"
                        }
                    }
                }
            }
        }
    }

    pipeline {
        agent any

        stages {
            stage("start") {
                steps {
                    echo "start"
                }
            }
            stage('parallel stage') {
                steps {
                    script {
                        echo "Inside script"
                        parallel parallelStagesMap
                    }
                }
            }
        }
    }

But this is the build result: Result The result stages are C1/C1/C1/C2/C2/C2/C3/C3/C3/C4/C4/C4 which means only the last parallel run is repeated 3 times.

Blue ocean Result

Do you have any suggestions or comments?

This is the console output:

    [Pipeline] Start of Pipeline
    [Pipeline] echo
    After value B1 at index 0
    [Pipeline] echo
    After value B2 at index 1
    [Pipeline] echo
    After value B3 at index 2
    [Pipeline] node
    Running on Jenkins in /Users/.jenkins/workspace/2D_array_test
    [Pipeline] {
    [Pipeline] stage
    [Pipeline] { (start)
    [Pipeline] echo
    start
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] { (parallel stage)
    [Pipeline] script
    [Pipeline] {
    [Pipeline] echo
    Inside script
    [Pipeline] parallel
    [Pipeline] { (Branch: B1)
    [Pipeline] { (Branch: B2)
    [Pipeline] { (Branch: B3)
    [Pipeline] stage
    [Pipeline] { (B1)
    [Pipeline] stage
    [Pipeline] { (B2)
    [Pipeline] stage
    [Pipeline] { (B3)
    [Pipeline] echo
    This is B1
    [Pipeline] sh
    [Pipeline] echo
    This is B2
    [Pipeline] sh
    [Pipeline] echo
    This is B3
    [Pipeline] sh
    [Pipeline] script
    [Pipeline] {
    [Pipeline] script
    [Pipeline] {
    [Pipeline] script
    [Pipeline] {
    [Pipeline] echo
    build_project_list = [A1, A2, A3, A4]
    [Pipeline] echo
    build_project_list = [B1, B2, B3, B4]
    [Pipeline] echo
    build_project_list = [C1, C2, C3, C4]
    [Pipeline] stage
    [Pipeline] { (C1)
    [Pipeline] stage
    [Pipeline] { (C1)
    [Pipeline] stage
    [Pipeline] { (C1)
    [Pipeline] echo
    C1
    [Pipeline] }
    [Pipeline] echo
    C1
    [Pipeline] }
    [Pipeline] echo
    C1
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] { (C2)
    [Pipeline] stage
    [Pipeline] { (C2)
    [Pipeline] stage
    [Pipeline] { (C2)
    [Pipeline] echo
    C2
    [Pipeline] }
    [Pipeline] echo
    C2
    [Pipeline] }
    [Pipeline] echo
    C2
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] { (C3)
    [Pipeline] stage
    [Pipeline] { (C3)
    [Pipeline] stage
    [Pipeline] { (C3)
    [Pipeline] echo
    C3
    [Pipeline] }
    [Pipeline] echo
    C3
    [Pipeline] }
    [Pipeline] echo
    C3
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] { (C4)
    [Pipeline] stage
    [Pipeline] { (C4)
    [Pipeline] stage
    [Pipeline] { (C4)
    [Pipeline] echo
    C4
    [Pipeline] }
    [Pipeline] echo
    C4
    [Pipeline] }
    [Pipeline] echo
    C4
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] }
    [Pipeline] }
    [Pipeline] }
    [Pipeline] // script
    [Pipeline] // script
    [Pipeline] // script
    [Pipeline] }
    [Pipeline] }
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] // stage
    [Pipeline] }
    [Pipeline] }
    [Pipeline] }
    [Pipeline] // parallel
    [Pipeline] }
    [Pipeline] // script
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] }
    [Pipeline] // node
    [Pipeline] End of Pipeline
    Finished: SUCCESS
jenkins-groovy parallel-processing jenkins groovy build
1 Answers

Try refactoring your code like shown below.

build_list = ['B1X', 'B2X', 'B3X']

l1 = ['A1', 'A2', 'A3', 'A4']
l2 = ['B1', 'B2', 'B3', 'B4']
l3 = ['C1', 'C2', 'C3', 'C4']

array_build_list = [ l1, l2, l3 ]

index = 1
index = -index

def parallelStagesMap = build_list.collectEntries {
    index++
    ["${it}" : generateStage(it, index)]
}

def generateStage(job, index) {
    return {
        stage("${job}") {
            echo "This is ${job}"
            script {
                build_project_list = array_build_list[index]
                echo "build_project_list = ${build_project_list}"
                runStages(build_project_list)
            }
        }
    }
}

def runStages(def build){
    for(int i=0; i<build.size(); i++) {
        stage(build[i]) {
            echo "Build ~~~~~ ${build[i]}"
        }
    }
    
}

pipeline {
    agent any

    stages {
        stage("start") {
            steps {
                echo "start"
            }
        }
        stage('parallel stage') {
            steps {
                script {
                    echo "Inside script"
                    parallel parallelStagesMap
                }
            }
        }
    }
}

enter image description here

Explanation

Basically, this has to do with the variable scopes. In your pipeline, the culprit is this line build_project_list = array_build_list[index] Since you don't have def in front of the variable declaration the build_project_list becomes a global variable that binds to the scope of the script. So when things execute in parallel all the stages will update this variable in parallel.

So there are two ways to fix this, method one is the one I added above, by scoping the variables by moving them to a new function. Method two is simply adding def key work to the variable build_project_list. Something like below.

def build_project_list = array_build_list[index]
ycr0
Related