k8s pods stuck in failed/shutdown state after preemption (gke v1.20)

user_865480 0
Viewed 2879

TL;DR - gke 1.20 preemptible nodes cause pods to zombie into Failed/Shutdown

We have been using GKE for a few years with clusters containing a mixture of both stable and preemptible node pools. Recently, since gke v1.20, we have started seeing preempted pods enter into a weird zombie state where they are described as:

Status: Failed

Reason: Shutdown

Message: Node is shutting, evicting pods

When this started occurring we were convinced it was related to our pods failing to properly handle the SIGTERM at preemption. We decided to eliminate our service software as a source of a problem by boiling it down to a simple service that mostly sleeps:

/* eslint-disable no-console */
let exitNow = false

process.on( 'SIGINT', () => {
  console.log( 'INT shutting down gracefully' )
  exitNow = true
} )

process.on( 'SIGTERM', () => {
  console.log( 'TERM shutting down gracefully' )
  exitNow = true
} )

const sleep = ( seconds ) => {
  return new Promise( ( resolve ) => {
    setTimeout( resolve, seconds * 1000 )
  } )
}

const Main = async ( cycles = 120, delaySec = 5 ) => {
  console.log( `Starting ${cycles}, ${delaySec} second cycles` )

  for ( let i = 1; i <= cycles && !exitNow; i++ ) {
    console.log( `---> ${i} of ${cycles}` )
    await sleep( delaySec ) // eslint-disable-line
  }

  console.log( '*** Cycle Complete - exiting' )
  process.exit( 0 )
}

Main()

This code is built into a docker image using the tini init to spawn the pod process running under nodejs (fermium-alpine image). No matter how we shuffle the signal handling it seems the pods never really shutdown cleanly, even though the logs suggest they are.

Another oddity to this is that according to the Kubernetes Pod logs, we see the pod termination start and then gets cancelled:

2021-08-06 17:00:08.000 EDT Stopping container preempt-pod

2021-08-06 17:02:41.000 EDT Cancelling deletion of Pod preempt-pod

We have also tried adding a preStop 15 second delay just to see if that has any effect, but nothing we try seems to matter - the pods become zombies. New replicas are started on the other nodes that are available in the pool, so it always maintains the minimum number of successfully running pods on the system.

We are also testing the preemption cycle using a sim maintenance event:

gcloud compute instances simulate-maintenance-event node-id

google-kubernetes-engine kubernetes node.js
2 Answers

After poking around various posts I finally relented to running a cronjob every 9 minutes to avoid the alertManager trigger that occurs after pods have been stuck in shutdown for 10+ minutes. This still feels like a hack to me, but it works, and it forced me to dig in to k8s cronjob and RBAC.

This post started me on the path: How to remove Kubernetes 'shutdown' pods

And the resultant cronjob spec:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-accessor-role
  namespace: default
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "delete", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-access
  namespace: default
subjects:
- kind: ServiceAccount
  name: cronjob-sa
  namespace: default
roleRef:
  kind: Role
  name: pod-accessor-role
  apiGroup: ""
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cronjob-sa
  namespace: default
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cron-zombie-killer
  namespace: default
spec:
  schedule: "*/9 * * * *"
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        metadata:
          name: cron-zombie-killer
          namespace: default
        spec:
          serviceAccountName: cronjob-sa
          restartPolicy: Never
          containers:
          - name: cron-zombie-killer
            imagePullPolicy: IfNotPresent
            image: bitnami/kubectl
            command:
              - "/bin/sh"
            args:
              - "-c"
              - "kubectl get pods -n default --field-selector='status.phase==Failed' -o name | xargs kubectl delete -n default 2> /dev/null"
status: {}

Note that the redirect of stderr to /dev/null is to simply avoid the error output from kubectl delete when the kubectl get doesn't find any pods in the failed state.

Update added missing "delete" verb from the role, and added the missing RoleBinding

Update added imagePullPolicy

user_8654800

Starting with GKE 1.20.5 and later, the kubelet graceful node shutdown feature is enabled preemptible nodes. From the note on the feature page:

When pods were evicted during the graceful node shutdown, they are marked as failed. Running kubectl get pods shows the status of the the evicted pods as Shutdown. And kubectl describe pod indicates that the pod was evicted because of node shutdown:

Status: Failed Reason: Shutdown Message: Node is shutting, evicting pods Failed pod objects will be preserved until explicitly deleted or cleaned up by the GC. This is a change of behavior compared to abrupt node termination.

These pods should eventually be garbage collected, although I'm not sure of the threshold value.

user_55297120
Related