How to cause an intentional restart of a single kubernetes pod

user_536874 0
Viewed 815

I am testing a log previous command and for that I need a pod to restart.

I can get my pods using a command like

kubectl get pods -n $ns -l $label

Which shows that my pods did not restart so far. I want to test the command:

kubectl logs $podname -n $ns --previous=true

That command fails because my pod did not restart making the --previous=true switch meaningless.

I am aware of this command to restart pods when configuration changed:

kubectl rollout restart deployment myapp -n $ns

This does not restart the containers in a way that is meaningful for my log command test but rather terminates the old pods and creates new pods (which have a restart count of 0).

I tried various versions of exec to see if I can shut them down from within but most commands I would use are not found in that container:

kubectl exec $podname -n $ns -- shutdown
kubectl exec $podname -n $ns -- shutdown now
kubectl exec $podname -n $ns -- halt
kubectl exec $podname -n $ns -- poweroff

How can I use a kubectl command to forcefully restart the pod with it retaining its identity and the restart counter increasing by one so that my test log command has a previous instance to return the logs from.

EDIT: Connecting to the pod is well described.

kubectl -n $ns exec --stdin --tty $podname -- /bin/bash

The process list shows only a handful running processes:

ls -1 /proc | grep -Eo "^[0-9]{1,5}$"

proc 1 seems to be the one running the pod. kill 1 does nothing, not even kill the proc with pid 1

I am still looking into this at the moment.

kubectl kubernetes
1 Answers

There are different ways to achieve your goal. I'll describe below most useful options.

Crictl

Most correct and efficient way - restart the pod on container runtime level.

I tested this on Google Cloud Platform - GKE and minikube with docker driver.

You need to ssh into the worker node where the pod is running. Then find it's POD ID:

$ crictl ps
CONTAINER           IMAGE               CREATED             STATE           NAME                ATTEMPT             POD ID
9863a993e0396       87a94228f133e       3 minutes ago       Running         nginx-3             2                   6d17dad8111bc

OR

$ crictl pods -s ready
POD ID              CREATED             STATE          NAME                    NAMESPACE       ATTEMPT         RUNTIME
6d17dad8111bc       About an hour ago   Ready          nginx-3                 default         2               (default)

Then stop it:

$ crictl stopp 6d17dad8111bc
Stopped sandbox 6d17dad8111bc

After some time, kubelet will start this pod again (with different POD ID in CRI, however kubernetes cluster treats this pod as the same):

$ crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                        ATTEMPT             POD ID
f5f0442841899       87a94228f133e       41 minutes ago      Running             nginx-3                     3                   b628e1499da41

This is how it looks in cluster:

$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
nginx-3                           1/1     Running   3          48m

Getting logs with --previous=true flag also confirmed it's the same POD for kubernetes.

Kill process 1

It works with most images, however not always.

E.g. I tested on simple pod with nginx image:

$ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
nginx            1/1     Running   0          27h

$ kubectl exec -it nginx -- /bin/bash
root@nginx:/# kill 1
root@nginx:/# command terminated with exit code 137

$ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
nginx            1/1     Running   1          27h

Useful link:

user_155372010
Related