How to retry a Jenkins Pipeline stage with an agent condition

Boris Katz 0
Viewed 57

in out company we execute parallel test executions when each test gets it's own node... The nodes are an aws spot type agents, which by policy can be taken by aws and get terminated while in progress. We want to re execute the stages which nodes were terminated by aws and we came across the Pipeline retry step retry-the-body-up-to-n-times step with condition, which can be set to retry a pipeline stage in case agent was interrupted. To use it we need to pass array of Nested Choice of Objects - in my case single object of "agent" i tried to execute the next code with different syntax attempts:

import hudson.model.*
import jenkins.model.*

pipeline {
    agent { label 'aws-fleet-spot-small' }
    stages {
        stage('test parallel') {
            steps {
                script {
                    def stepsForParallel = [:]
                    stepsForParallel["stage1"] = {
                        retry(count: 3, conditions: [agent{ label 'aws-fleet-spot-small' }]){
                            node("aws-fleet-spot-small") {
                                stage("stage1") {
                                    script {
                                        echo ('stage11111')
                                    }
                                }
                            }
                        }
                    }
                    stepsForParallel["stage2"] = {
                        node("aws-fleet-spot-small") {
                            stage("stage2") {
                                script {
                                    echo ('stage2222')
                                }
                            }
                        }
                    }
                    parallel stepsForParallel
                }
            }
        }
    }
}

or

retry(count:3, conditions:[agent])

i also found this stuck overflow thread Retry with condition jenkins pipeline and use the code :

import org.jenkinsci.plugins.workflow.support.steps.AgentErrorCondition
retry(count: 3, conditions: [new AgentErrorCondition()){

or

retry(count:3, conditions: [AgentErrorCondition])

which by the jenkins code, expects ErrorCondition implementing classes arr/list as a value to the conditions key but i get compilation exceptions

Can anyone please suggest me the right syntax or point to a working example?

jenkins-pipeline jenkins
1 Answers

In our company we have exactly the same setup, and here's the code that retries the step when the AWS node was taken away or some other problem happened on that node:

def class PipelineHelper implements Serializable {
    def steps

    PipelineHelper(steps) { this.steps = steps }

    void retry(final Closure<?> action, int maxAttempts, final int count = 0) {
        steps.echo "Trying action, attempt count is: ${count}"
        try {
            action.call();
        } catch (final x) {
            steps.echo "Exception: ${x.toString()}"
            def x_causes = [x.getClass()]
            def exc = x
            while (exc.getCause()) {
                exc = exc.getCause()
                x_causes += exc.getClass()
            }
            steps.echo "Causes: ${x_causes.toString()}"
            
            def allowed_causes = [
                // Pipeline stopped org.jenkinsci.plugins.workflow.steps.FlowInterruptedException,
                org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution.RemovedNodeCause,
                org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException,//Unable to create live FilePath
                hudson.remoting.ChannelClosedException,
                java.nio.file.FileSystemException, // no space left on device
                java.io.IOException,
            ]
            
            if (x_causes.intersect(allowed_causes)) {
                if (count <= maxAttempts) {
                    steps.sleep(10)
                    steps.echo "Retrying from failed stage."
                    return retry(action, maxAttempts, count + 1)
                } else {
                    steps.echo "Max attempts reached. Will not retry."
                    throw x
                }
            } else {
                steps.echo 'Aborting'
                throw x
            }
        }
    }
}

You then wrap your code in the following manner:

                    def pipelineHelper = new pipelineHelper(this)
                    pipelineHelper.retry( {
                        node("aws-fleet-spot-small") {
                            echo ('stage11111')
                        }
                    }, MAX_RETRIES)
MaratC0
Related