Jenkins job stuck on "Pausing (Preparing for shutdown)" after entering "quietDown" mode

user_777961 0
Viewed 1337

I'm running Jenkins in AWS and I'm writing a script that safely re-deploys the Jenkins CloudFormation stack. I'm not trying to simply restart the Jenkins web server. I know about the different "safe" options, but none of them seem to work for me:

  • http://<jenkins.server>/safeRestart - this seems to not work for me, since it just restarts the Jenkins web server. The jobs that were running before restart continue to run normally after restart. I'm trying to terminate the AWS instance.
  • http://<jenkins.server>/safeExit - allows current jobs to finish, but immediately kills the Jenkins web server and there is no way for me to check when it's safe to terminate the AWS instance.
  • http://<jenkins.server>/quietDown - this almost does exactly what I want. New jobs are queued, not started (good). The Jenkins web server is up and I can query for running jobs (good). However, running jobs go into this weird state, where upon job completion, the job doesn't actually exit. The console hangs with Pausing (Preparing for shutdown). Because of this, I can't query for the number of the currently running jobs, since it picks up all the jobs in this weird state.

What are my options here? Is there something wrong with my setup? Is there a way to prevent jobs from entering Pausing (Preparing for shutdown) when Jenkins is in "quietDown" mode, and the job finishes?

jenkins-pipeline jenkins
1 Answers

The"quietDown" mode, unfortunately PAUSES all currently running Workflow (Pipeline) jobs making it an unpleasant alternative. The pausing behavior is "by design".

There is a plugin called "Lenient-ShutDown" that works better than the shutdown plugin for this purpose. But it has caveats in that it's not able to fully stop all jobs from starting. One of it's strengths, however is that you can allowlist certain jobs (e.g. folders) so that those jobs are not prevented from starting.

These two plugins together with some monitoring will allow you to accomplish your goal of safely restarting. One way is via a simple external script that doesn't do any monitoring at all, just pauses... Using the SSH cli is the easiest way to do this, because you can mix in groovy code to do other things, for example our process looks like this:

E.g. (BASH)

sleeptime()  { if [ "$1" -ne "0" ]; then echo "Sleeping for $1 seconds..."; sleep $1; fi }

MYJENKINS='MYJENKINS_DNS -p PORTNO'
ssh $MYJENKINS lenient-quiet-down
sleeptime 500

ssh $MYJENKINS quiet-down
sleeptime 200

# OPTIONAL a groovy script that will either wait until no jobs, or kill all running jobs
ssh $MYJENKINS groovy = < KILLallJobs.groovy

ssh $MYJENKINS restart

Obviously if you're going through the trouble of running a groovy script, you can do all of these things in that one script.

user_16557800
Related