Kubernetes pod fail/restart simulation

Question

We have a data visualization server hosted in Kubernetes pods. The dashboards in that data viz are displayed in the browser of different monitors/terminals for near-real time operational reporting. Sometimes the pods fail, and when they come alive again, the browser redirects to Single Sign-On page instead of going to the dashboard the URL is originally configured to.

The server are hosted in I would presume a replica set. There are two pods that exist as far as I can tell.

I was granted privilege on using kubectl to solve this problem, but still quite new with the whole Kubernetes thing. Using kubectl, how do I simulate pod failure/restart for testing purposes? Since the pods are in duplicate, shutting one of them will only redirect the traffic to the other pod. How to make both pods fail/restart at the same time? (I guess doing kubectl delete pod on both pods will do, but I want to make sure k8s will respawn the pods automatically, and not delete them forever).

user_6309601 · Answer

If I understand the use case correctly, you might want to use kubectl scale command. This will give you the flexibility to make the replica count to zero to N by running a simple kubectl scale command. See examples. Also, if you are using deployment, you can just do the kubectl delete pod, the deployment controller will spawn a new one to satisfy the replica count.

kubectl scale deployment/  --replicas=

short example:

kubectl scale deployment/deployment-web  --replicas=0
deployment.apps/deployment-web scaled

Long Example:

// create a deployment called, deployment-web with two replicas.

kubectl create  deployment  deployment-web --image=nginx --replicas 2
deployment.apps/deployment-web created

// verify that both replicas are up

kubectl get deployments.apps 
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
deployment-web   2/2     2            2           13s

// expose the deployment with a service [OPTIONAL-STEP, ONLY FOR EXPLANATION]

kubectl expose deployment deployment-web  --port 80
service/deployment-web exposed

//verify that the service is created

kubectl get svc
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
deployment-web   ClusterIP   10.233.24.174           80/TCP    5s

// dump the list of end-points for that service, there would be one for each replica. Notice the two IPs in the 2nd column.

kubectl get ep
NAME             ENDPOINTS                         AGE
deployment-web   10.233.111.6:80,10.233.115.9:80   12s

//scale down to 1 replica for the deployment

kubectl scale --current-replicas=2 --replicas=1 deployment/deployment-web
deployment.apps/deployment-web scaled

// Notice the endpoint is reduced from 2 to 1.

kubectl get ep
NAME             ENDPOINTS           AGE
deployment-web   10.233.115.9:80     43s

// also note that there is only one pod remaining

kubectl get pod
NAME                             READY   STATUS    RESTARTS   AGE
deployment-web-64c769b44-qh2qf   1/1     Running   0          105s

// scale down to zero replica

kubectl scale --current-replicas=1 --replicas=0 deployment/deployment-web
deployment.apps/deployment-web scaled

// The endpoint list is empty

kubectl get ep
NAME             ENDPOINTS           AGE
deployment-web                 9m4s

//Also, both pods are gone

kubectl get pod 
No resources found in default namespace.

// When you are done with testing. restore the replicas

kubectl scale --current-replicas=0 --replicas=2 deployment/deployment-web
deployment.apps/deployment-web scaled

//endpoints and pods are restored back

kubectl get ep
NAME             ENDPOINTS                          AGE
deployment-web   10.233.111.8:80,10.233.115.11:80   10m
foo-svc          10.233.115.6:80                    50m
kubernetes       192.168.22.9:6443                  6d23h


kubectl get pod -l app=deployment-web
NAME                             READY   STATUS    RESTARTS   AGE
deployment-web-64c769b44-b72k5   1/1     Running   0          8s
deployment-web-64c769b44-mt2dd   1/1     Running   0          8s

Kubernetes pod fail/restart simulation

1 Answers