I am trying to implement a graceful shutdown on an elastic beanstalk application and I have put together this shutdown test-
Main-
public class Main {
public static void main(String[] args) throws InterruptedException {
ShutdownTest shutdownTest = new ShutdownTest();
shutdownTest.run();
System.out.println("END OF MAIN");
}
}
Test Class-
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class ShutdownTest {
private volatile long jobEnd;
private volatile boolean shutdown = false;
public void run() throws InterruptedException {
Thread mainThread = Thread.currentThread();
Thread shutdownHook = new Thread(() -> {
runShutdownTimer();
try {
System.out.println("Got shutdown signal.");
shutdown = true;
mainThread.join();
} catch(Exception e) {
System.out.println(e);
}
});
Runtime.getRuntime().addShutdownHook(shutdownHook);
while(!shutdown) {
long jobStart = System.currentTimeMillis();
int jobRunTime = 60000;
jobEnd = jobStart + jobRunTime;
Thread.sleep(jobRunTime);
}
System.out.println("End of Run");
}
public void runShutdownTimer() {
long start = System.currentTimeMillis();
float waitTime = (jobEnd - start)/1000f;
System.out.println("Shutdown phase should take " + waitTime + " seconds");
ExecutorService executorService = Executors.newFixedThreadPool(1);
executorService.execute(() -> {
while(true) {
long now = System.currentTimeMillis();
float diff = (float)(now - start)/1000f;
System.out.println(diff + " seconds elapsed in shutdown phase");
try {
Thread.sleep(300);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
}
}
If I run this locally and send a shutdown signal (^C or stopping execution through my IDE) I will get an output like-
Shutdown phase should take 45.139 seconds
Got shutdown signal.
0.006 seconds elapsed in shutdown phase
0.311 seconds elapsed in shutdown phase
0.615 seconds elapsed in shutdown phase
...
44.433 seconds elapsed in shutdown phase
44.734 seconds elapsed in shutdown phase
45.035 seconds elapsed in shutdown phase
End of Run
END OF MAIN
<END OF EXECUTION>
But if I run this on Elastic Beanstalk and send a shutdown signal by pushing a new version I will get something like this-
Shutdown phase should take 25.647 seconds
Got shutdown signal.
0.007 seconds elapsed in shutdown phase
0.307 seconds elapsed in shutdown phase
0.607 seconds elapsed in shutdown phase
...
9.614 seconds elapsed in shutdown phase
9.915 seconds elapsed in shutdown phase
10.215 seconds elapsed in shutdown phase
<END OF EXECUTION>
I've done this a couple time and it seems the execution forcefully quits at around 10 seconds regardless of how much time it should wait. The jobs these applications run may take longer than 10 seconds, relying on network and request speeds of outside forces that aren't controllable.
Is there a way to increase the forced termination timeout?
Edit:
I've tried sshing into the box and killing the process through pkill java (without signal). The logs show the application went through a 28 second shutdown process successfully. Something about the elastic beanstalk service is issuing a second sigterm as I suspected. I'll updated when i figure out what it is.
Edit Edit:
I looked into ASG lifecycle hooks. These work by emitting a cloudwatch event when a termination event starts. This event contains the ec2 instance id and could be sent-to/routed-by an SNS topic. When ASG lifecycle hooks are configured to emits events like this, it will wait for a confirmation that the termination can continue with a configurable timeout period. This means each EC2 in the ASG can listen to that SNS topic for its own termination event, start the shutdown procedure, then alert ASG that the termination process can continue.
This could work for scaling, but this won't work when updating the EC2 environment- causing an apprestart.