" Pod is blocking scale down because it has local storage "

Types of pods that prevent Cluster Autoscaler from removing a node:

• Pods with restrictive PodDisruptionBudget.

• Kube-system pods that:

  are not run on the node by default
  
  don't have a pod disruption budget set or their PDB is too restrictive (since CA 0.6).
  

• Pods that are not backed by a controller object (so not created by deployment, replica set, job, stateful set etc). *

• Pods with local storage. *

• Pods that cannot be moved elsewhere due to various constraints (lack of resources, non-matching node selectors or affinity, matching anti-affinity, etc)

• Pods that have the following annotation set:

"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

So, your Pod is blocking scale down because it requests local storage. You should add the annotation(supported in CA 1.0.3 or later) to the pod:

“cluster-autoscaler.kubernetes.io/safe-to-evict": "true”

For more information refer to the Cluster Autoscaler FAQs.

You can also refer to the following stack overflow case which is similar to your case.

You need to add the once annotation to POD so that cluster autoscaler can remove that POD from POD safe to evict.

cluster-autoscaler.kubernetes.io/safe-to-evict": "true"

above annotation, you have to add in into POD.

You can read more at : https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility#cluster-not-scalingdown

NoScaleDown example: You found a noScaleDown event that contains a per-node reason for your node. The message ID is "no.scale.down.node.pod.has.local.storage" and there is a single parameter: "test-single-pod". After consulting the list of error messages, you discover this means that the "Pod is blocking scale down because it requests local storage". You consult the Kubernetes Cluster Autoscaler FAQ and find out that the solution is to add a "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation to the Pod. After applying the annotation, cluster autoscaler scales down the cluster correctly.

For further clarification, you can use this command to update the pod's annotation:

kubectl annotate pod <podname> -n <namespace> "cluster-autoscaler.kubernetes.io/safe-to-evict=true"

Had the same error when using Gitlab + Autodevops + GoogleCloud. The issue is the cm_acme pods's that are spun up to answer the letsencrypt challenges.

e.g. we have pods like this cm-acme-http-solver-d2tak
hanging around in our cluster so the cluster won't downsize until these pods are destroyed.

A simple

kubectl get pods -A | grep cm-acme

will list all the pods that need to be destroyed with

kubectl delete pod -n {namespace} {pod name}