How to make sure Kubernetes autoscaler not deleting the nodes which runs specific pod

user_4993583 0
Viewed 10323

I am running a Kubernetes cluster(AWS EKS one) with Autoscaler pod So that Cluster will autoscale according to the resource request within the cluster.

Also, cluster will shrink no of nodes when the load is reduced. As I observed, Autosclaer can delete any node in this process.

I want to control this behavior such as asking Autoscaler to stop deleting nodes that runs a specific pod. For example, If a node runs the Jenkins pod, Autoscaler should skip that node and delete other matching node from the cluster.

Will there a way to achieve this requirement. Please give your thoughts.

aws-auto-scaling kubernetes
3 Answers

You can use "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

...

template:
     metadata:
       labels:
         app: jenkins
       annotations:
         "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

     spec:
       nodeSelector:
         failure-domain.beta.kubernetes.io/zone: us-west-2b
...
user_38495550

You should set a pod disruption budget that references specific pods by label. If you want to ensure that there is at least one Jenkins worker pod running at all times, for example, you could create a PDB like

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: jenkins-worker-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: jenkins
      component: worker

(adapted from the basic example in Specifying a Disruption Budget in the Kubernetes docs).

Doing this won't prevent nodes from being destroyed; the cluster autoscaler is still free to scale things down. What it will do is temporarily delay destroying a node until the disruption budget can be met again.

For example, say you've configured your Jenkins setup so that there are three workers. Two get scheduled on the same node, and the autoscaler takes that node offline. The ordinary Kubernetes Deployment system will create two new replicas on nodes that still exist. If the autoscaler also decides it wants to destroy the node that has the last worker, the pod disruption budget above will prevent it from doing so until at least one other worker is running.

When you say "the Jenkins pod" in the question, there are two other important implications to this. One is that you should almost always configure your applications using higher-level objects like Deployments or StatefulSets and not bare Pods. The other is that it is generally useful to run multiple copies of things for redundancy if nothing else. Even absent the cluster autoscaler, disks fail, Amazon occasionally arbitrarily decommissions EC2 instances, and nodes otherwise can go offline outside of your control; you often don't want just one copy of something running in your cluster, especially if you're considering it a critical service.

user_100081730

In autoscaler FAQ on github you can read the following:

What types of pods can prevent CA from removing a node?

  • Pods with restrictive PodDisruptionBudget.
  • Kube-system pods that:
    • are not run on the node by default, *
  • Pods that are not backed by a controller object (so not created by deployment, replica set, job, stateful set etc). *
  • Pods with local storage. *
  • Pods that cannot be moved elsewhere due to various constraints (lack of resources, non-matching node selectors or affinity, matching anti-affinity, etc)
  • Pods that have the following annotation set: "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

*Unless the pod has the following annotation (supported in CA 1.0.3 or later): "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"

user_122010840
Related