Kubernetes multi-container pod termination process

user_8118147 0
Viewed 958

I have a multi-container pod in k8s, let's call them A and B. When stopping the pod, A must stop before B because A needs B until it's off.

To do that, I registered a preStop hook on A so A can gracefully stop before B.

However I'm not sure this is a good solution, because I miss some information I can't find in k8s documentation:

What happens when a multi-container pod is stopped?

  • All containers preStop hooks are called, then when they are all over all containers receive SIGTERM, or
  • In parallel, all containers receive preStop if they have one or directly SIGTERM if they don't?

In the second case, preStop is useless for what I want to do as B will be instantly killed.

kubernetes-pod kubernetes termination
1 Answers

Typically, during pod deletion, the container runtime sends a TERM signal to the main process in each container.

According to the official documentation:

  1. If one of the Pod's containers has defined a preStop hook, the kubelet runs that hook inside of the container.

  2. The kubelet triggers the container runtime to send a TERM signal to process 1 inside each container.

This numeration can confuse - looks like TERM signal will be sent only after preStop hook will be finished. I decided to check the order of work with a simple example below.

apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-demo
spec:
  restartPolicy: Never
  volumes:
  - name: config
    configMap:
      name: nginx-conf
  containers:
  - name: container-1
    image: nginx
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sleep","15"]
    ports:
    - containerPort: 80
  - name: container-2
    image: nginx
    ports:
    - containerPort: 81
    volumeMounts:
    - name: config
      mountPath: /etc/nginx/conf.d
  terminationGracePeriodSeconds: 30

Container-1 has preStop hook for 15 seconds delay. I've connected to both containers to see behavior during pod deletion.

Result

After pod deletion:

  1. Container-1 worked for 15 seconds, before the connection was lost

  2. Container-2 immediately lost connection

Conclusion

If the container has a preStop hook, it will try to execute it. Only then it will receive TERM signal. The main condition in this case: the grace period has not expired.

If the container doesn't have a preStop hook, it will receive TERM signal immediately after the command to remove the pod. Thus, it will not wait whilepreStop hook will be executed for another container.

Note: The containers in the Pod receive the TERM signal at different times and in an arbitrary order. If the order of shutdowns matters, consider using a preStop hook to synchronize.

user_165592920
Related