Kubernetes graceful shutdown of a Spring Boot Tomcat application

user_11951718 0
Viewed 5792

I am running a Java Spring Boot boot application on Tomcat in a Kubernetes cluster, I am using a lifecycle hook to shut down Tomcat before the container gets terminated. Also defining a termination grace period of 30 secs.

Expected behaviour: tomcat shuts down gracefully when I kill a pod or delete a deployment.

Actual behaviour: tomcat shuts down abruptly.

lifecycle:
          preStop:
            exec:
              command:
              - /usr/local/tomcat/bin/catalina.sh

Can anyone help please?

kubernetes tomcat
2 Answers

In Kubernetes, the preStop hook is executed before the pod is terminated.

For the above use case, tomcat process has to be shutdown gracefully before the pod is terminated when a pod / deployment is deleted.

The following sample pod definition works, by graceful shutdown of the tomcat server, before pod termination. Note: The container in this definition is FROM tomcat:9.0.19-jre8

apiVersion: v1 
kind: Pod 
metadata:
  name: demopod
spec:
  containers:
  - image: demo:web
    name: demo-container
    ports:
    - containerPort: 8080
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
      preStop:
        exec:
          command: ["/bin/sh", "-c", "echo Stopping server > /usr/share/message1; sleep 10; sh /usr/local/tomcat/bin/catalina.sh stop"]

In the above definition file, we run three commands (change as per need).

  1. Write a shutdown message into /usr/share/message1

  2. sleep 10 (Just to give time to view pod / container logs)

  3. Run Catalina.sh stop script to stop the tomcat server.

To Debug: For PreStop, if the handler fails, an event is broadcasted, the FailedPreStopHook event. You can see these events by running kubectl describe pod <pod_name>

Refer below post for more detailed instructions: http://muralitechblog.com/kubernetes-graceful-shutdown-of-pods/

Shutdown logs: Valid shutdown logs

user_26559140

This is a common mistake about graceful period concept. People think if they set a graceful period of say 30 seconds, when deleting the pod, it is going to wait for 30 seconds, which is wrong.

The grace period is actually for killing a pod, not to keep it alive. If you have an app that shuts down in 1 second, you can set any grace period, but the pod is going to die in 1 second. It would kill the pod (by sending SIGKILL), if it passes the grace period.

So, you would set a grace period to not have a pod in terminating state forever (and kill it after the certain amount of time), rather then keep the pod running until certain amount of time after deleting it. Hope this makes sense.

And you preStop probably runs, if everything is set properly, but it will depend on what are you doing to see the results. You can't see logs, for example, unless you have made them persistent.

user_55645780
Related