Kubernetes node affinity behavior

user_486478 0
Viewed 75

I'm trying to affect pods to a specific node using affinities but it ends with a strange behavior I can't understand.

Let explain my nodes setup. I have "x" nodes which all have the labels kali=true, two nodes have in addition the labels kali-app=true and one of these nodes have the label kali-app-1=true.

Now I try to deploy a Deployment with a replicas of 2 pods which should result with putting these pods on the kali-app-1 node:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kali
              operator: In
              values: ['true']
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
            - key: kali-app
              operator: In
              values: ['true']
      - weight: 2
        preference:
          matchExpressions:
            - key: kali-app-1
              operator: In
              values: ['true']

The result is the first pod is put on the kali-app-1 node but the second one is put on the node which only have the kali-app node (namely kali-app-2=true which is an other label I have in my cluster).

Does anyone can explain me this behavior?

kubernetes
1 Answers

Based on the nodeAffinity documentation it works as expected since preferredDuringSchedulingIgnoredDuringExecution field is used for the 2nd and 3rd conditions.

They way it works is:

There are currently two types of node affinity, called requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution. You can think of them as "hard" and "soft" respectively, in the sense that the former specifies rules that must be met for a pod to be scheduled onto a node (similar to nodeSelector but using a more expressive syntax), while the latter specifies preferences that the scheduler will try to enforce but will not guarantee.

The weight field in preferredDuringSchedulingIgnoredDuringExecution is in the range 1-100. For each node that meets all of the scheduling requirements (resource request, RequiredDuringScheduling affinity expressions, etc.), the scheduler will compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node matches the corresponding MatchExpressions. This score is then combined with the scores of other priority functions for the node. The node(s) with the highest total score are the most preferred.

That means that after first pod is scheduled on the required node, another node is more preferred for placing the second pod. (e.g. if the deployment is heavy and has high cpu/memory requests) + generally kubernetes scheduler tries to spread pods across nodes to comply with high availability. Please get familiar with NodeAffinity preferredDuringSchedulingIgnoredDuringExecution doesn't work well GitHub issue.

Therefore if these pods from the deployment have to be scheduled on the same one node with label kali-app-1, you need to use requiredDuringSchedulingIgnoredDuringExecution field to enforce this assignment and do not provide the scheduler with other possible options.

user_155372010
Related