How to create dynamic jenkins pipeline with separate container for every stages?

user_2458471 0
Viewed 75

I have static working jenkins pipelines:

pipeline {
  agent none
  options {
    buildDiscarder(logRotator(daysToKeepStr: '7'))
    timeout(time: 90, unit: 'MINUTES')
  }
  stages {
    stage ('1. Prepare') {
      agent {
        kubernetes {
          yaml agentPod()
          defaultContainer 'agent'
        }
      }
      steps {
        script {
          echo "Prepare pipeline"
        }
      }
    }
    stage ('2. Check') {
      parallel {
        stage('Runs on master') {
          agent {
            kubernetes {
              yaml agentPod("500m", "500Mi", "100m", "200Mi")
              defaultContainer 'agent'
            }
          }
          steps {
            echo "Running on master"
          }
        }
        stage('Runs on agent1') {
          agent {
            kubernetes {
              yaml agentPod("500m", "500Mi", "100m", "200Mi")
              defaultContainer 'agent'
            }
          }
          steps {
            echo "Running on agent1"
          }
        }
        stage('Runs on agent2') {
          agent {
            kubernetes {
              yaml agentPod("500m", "500Mi", "100m", "200Mi")
              defaultContainer 'agent'
            }
          }
          steps {
            echo "Running on agent2"
          }
        }
      }
    }
  }
}

but I need to create the same pipeline but use a dynamic function. I try to use several ideas, but every time it doesn't work. Very important is that for every stage I need create a separate kubernetes pod (with kaniko image to build new image). I try to use something like this but it desn't work.

def generateStage(podLabel) {
  return {
    agent {
      kubernetes {
        yaml agentPod("500m", "500Mi", "100m", "200Mi")
        defaultContainer 'agent'
      }
    }
    steps {
      echo "Running on ${podLabel}"
    }
  }
}

def parallelStagesMap = [:]

pipeline {
  agent none
  options {
    buildDiscarder(logRotator(daysToKeepStr: '7'))
    timeout(time: 90, unit: 'MINUTES')
  }
  stages {
    stage ('1. Prepare') {
      agent {
        kubernetes {
          yaml agentPod()
          defaultContainer 'agent'
        }
      }
      steps {
        script {
          def agents = ['master', 'agent1', 'agent2']
          parallelStagesMap = agents.collectEntries {
            ["Runs on ${it}" : generateStage(it)]
          }
          echo "Prepare pipeline"
        }
      }
    }
    stage ('2. Check') {
      parallel parallelStagesMap 
    }
  }
}

because after I execute this code I see this error:

org.jenkinsci.plugins.workflow.cps.CpsCompilationErrorsException: startup failed:

/var/jenkins_home/jobs/app/jobs/pipeline/branches/PR-14476/builds/24/libs/e647be597f45b6129772d69874a82199dfce9ad821d01ce80c7a153b2c310c04/vars/dashboardPipelineTemplate.groovy: 46: Expected a block for parallel @ line 46, column 7.
         parallel parallelStagesMap 
         ^

/var/jenkins_home/jobs/app/jobs/pipeline/branches/PR-14476/builds/24/libs/e647be597f45b6129772d69874a82199dfce9ad821d01ce80c7a153b2c310c04/vars/dashboardPipelineTemplate.groovy: 46: No stages specified @ line 46, column 7.
         parallel parallelStagesMap 
         ^

/var/jenkins_home/jobs/app/jobs/pipeline/branches/PR-14476/builds/24/libs/e647be597f45b6129772d69874a82199dfce9ad821d01ce80c7a153b2c310c04/vars/dashboardPipelineTemplate.groovy: 46: No stages specified @ line 46, column 7.
         parallel parallelStagesMap 
         ^

Does anyone have any idea how to build something like this correctly?

jenkins-plugins jenkins
2 Answers

Once you move into Scripted Pipeline Syntax you can no longer use Declarative Pipeline Syntax like agent. So instead of using the agent directive try using Pod Template step. Something like this. Your generateStage(podLabel) function should look something like the one below.

def generateStage(podLabel) {
  return {
    stage(podLabel) {
        podTemplate {
            node(POD_LABEL) {
                // pipeline steps...
            }
        }
    }
  }
}
user_26270180

This is an answer to how to correctly create a dynamic parallel Jenkins pipeline with separate container. I added an extra condition to show you how to add "extra" logic to create an extra condition, that not all elements in the list should proceed.

def generateStage(podLabel) {
  return {
    stage("Runs on ${podLabel}") {
      podTemplate(yaml: agentPod("500m", "500Mi", "100m", "200Mi")){
        node(POD_LABEL) {
          echo "Running on ${podLabel}"
        }
      }
    }
  }
}

def parallelStagesMap = [:]

pipeline {
  agent none
  options {
    buildDiscarder(logRotator(daysToKeepStr: '7'))
    timeout(time: 90, unit: 'MINUTES')
  }
  stages {
    stage ('1. Prepare') {
      agent {
        kubernetes {
          yaml agentPod()
          defaultContainer 'agent'
        }
      }
      steps {
        script {
          def agents = ['master', 'agent1', 'agent2']
          agents.each{
            if (it != "master") {
              parallelStagesMap["Runs on ${it}"] = generateStage(it)
            }
          }
          echo "Prepare pipeline"
        }
      }
    }
    stage ('2. Check') {
      steps {
        script {
          parallel parallelStagesMap 
        }
      }
    }
  }
}
user_24584710
Related