Jenkins declarative pipeline waits for executor indefinitely

Question

I'm no Jenkins expert so please excuse me if this is an absolute rookie question.

I'm currently investigating a related issue with our Jenkins pipeline. While attempting to provision an executor from AWS EC2, the connection timed out. I'd expect to the job to abort after a few retries. Instead it keeps trying to connect. The connection keeps timing out. Infinite loop!

A timeout is defined in the pipeline. As there is no executor, the timeout is never read.

pipeline {

   agent {
       label 'docker'
   }

    options {
         timeout(time: 45, unit: 'MINUTES')
         timestamps()
         skipDefaultCheckout()
         disableConcurrentBuilds()
         buildDiscarder(logRotator(numToKeepStr:'5'))
    }

    stages {
        // And so on...

The 'Build Timeout' plugin is installed on our Jenkins, but it doesn't seem to be active for this project, so I don't see the settings for aborting the project.

Installed plugins

Amazon EC2, 1.42
Build Timeout 1.19
Pipeline 2.6

Note that the pipeline has worked in the past.

[EDIT]

The following log has not changed in days.

Started by user a.user
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to http://sources:8080/scm/git/backend
 > git config remote.origin.url http://sources:8080/scm/git/backend # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials provides read-only access to the project git repositories
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/branch1
Seen branch in repository origin/branch2
Seen branch in repository origin/branch3
Seen branch in repository origin/branch4
Seen branch in repository origin/branch5
Seen branch in repository origin/branch6
Seen branch in repository origin/branch7
Seen branch in repository origin/branch8
Seen branch in repository origin/branch9
Seen branch in repository origin/branch10
Seen branch in repository origin/branch11
Seen branch in repository origin/branch12
Seen branch in repository origin/branch13
Seen 13 remote branches
Obtained ci/jobs/Backend-Build-Multibranch/Jenkinsfile from d6d39ddab89bb77502c83f99c7f08f0b6eb03e77
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
Waiting for next available executor

user_7571258 · Answer

To set a timeout for agent connection, you can define agent { label 'docker' } for a stage instead of the whole pipeline. Inside this stage you can create nested stages for the stages you previously had defined at the top level, so they'll all run on the "docker" node when it is available.

pipeline {
    agent{ label 'master' }

    options {
        timeout(time: 45, unit: 'MINUTES')
    }    

    stages {
        stage('Connect Node') {
            agent { label 'docker' }

            stages {
                stage('Build') {
                    steps{
                        echo 'Hello'
                    }
                }
                // And so on...
            }
        }
    }
}

user_5767497 · Answer

Here are the modifications to https://stackoverflow.com/users/7571258/zett42 's answer:

pipeline {
   // 'agent none' prevents Jenkins from starting the node. 'agent' must be
   // defined in subsequent stages.
   agent none

    // Options are read and applied globally.
    options {
         timeout(time: 45, unit: 'MINUTES')
    }

    stages {
        // Node connection stage will time out as declared in the options block
        stage('Connect Node') {
            agent {
                label 'docker'
            }

            stages {
                // And so on...
            }
        }
    }
}

user_9442694 · Answer

What is exact messages you are seeing for executor waiting? It might be that there are no slaves running with label 'docker' when the job is triggered and/or new slave is also not being triggered (could be due to static slave count for AMI/ec2 instances cap limit or no dynamic slave spin up)

Jenkins declarative pipeline waits for executor indefinitely

3 Answers