What is topologyKey in pod affinity?

user_2237469 0
Viewed 1687

I cannot really understand the purpose and usage of topologyKey in pod affinity. The documentations says:

topologyKey is the key of node labels. If two Nodes are labelled with this key and have identical values for that label, the scheduler treats both Nodes as being in the same topology. The scheduler tries to place a balanced number of Pods into each topology domain.

And example usage is as follows:

kind: Pod
metadata:
  name: with-pod-affinity
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: security
            operator: In
            values:
            - S1
        topologyKey: topology.kubernetes.io/zone
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: security
              operator: In
              values:
              - S2
          topologyKey: topology.kubernetes.io/zone
  containers:
  - name: with-pod-affinity
    image: k8s.gcr.io/pause:2.0

So where does topology.kubernetes.io/zone come from? How can I know what value should I provide for this topologyKey field in my yaml file, and what happens if I just put a random string here? Should I label my node and use the key of this label in topologyKey field?

Thank you.

kubernetes
3 Answers

A topology Key is effectively just a label that you assign to your nodes or that a cloud provider has already assigned.

The intent is to indicate certain topology characteristics, like the availability zone or server rack, for example. But they are actually arbitrary.

It is documented here.

For example, you want to spread out pods across 3 different availability zones. The topology key could help you to achieve this, as it prevents them from being randomly scheduled in the same zone.

Here are 2 examples from the docs:

For example, you could use requiredDuringSchedulingIgnoredDuringExecution affinity to tell the scheduler to co-locate Pods of two services in the same cloud provider zone because they communicate with each other a lot. Similarly, you could use preferredDuringSchedulingIgnoredDuringExecution anti-affinity to spread Pods from a service across multiple cloud provider zones.

user_92088870

Required as part of a affinity.podAffinity or affinity.podAntiAffinity spec section, the topologyKey field is used by the scheduler to determine the domain for Pod placement.

The topologyKey domain is used to determine relative placement of the Pods being scheduled relative to the Pods identified by the ...labelSelector.matchExpressions section.

With podAffinity, a Pod will be scheduled in the same domain as the Pods that match the expression.

Two common label options are topology.kubernetes.io/zone and kubernetes.io/hostname. Others can be found in the Kubernetes Well-Known Labels, Annotations and Taints documentation.

  • topology.kubernetes.io/zone: Pods will be scheduled in the same zone as a Pod that matches the expression.
  • kubernetes.io/hostname: Pods will be scheduled on the same hostname as a Pod that matches the expression.

For podAntiAffinity, the opposite is true: Pods will not be scheduled in the same domain as the Pods that match the expression.

The Kubernetes documentation Assigning Pods to Nodes documentation (Inter-pod affinity and anti-affinity section) provides a additional explanation.

user_2952900

topology.kubernetes.io/zone will be set only if you are using a cloudprovider. However, you should consider setting this on nodes if it makes sense in your topology.

it is documented here.

# This is an example of the values from an AWS cluster
❯ k get nodes --show-labels | awk '{print $6}' | tr ',' '\n' | grep topology
topology.kubernetes.io/region=eu-central-1
topology.kubernetes.io/zone=eu-central-1a
topology.kubernetes.io/region=eu-central-1
topology.kubernetes.io/zone=eu-central-1a
topology.kubernetes.io/region=eu-central-1
topology.kubernetes.io/zone=eu-central-1a
topology.kubernetes.io/region=eu-central-1
topology.kubernetes.io/zone=eu-central-1a
topology.kubernetes.io/region=eu-central-1
topology.kubernetes.io/zone=eu-central-1b
topology.kubernetes.io/region=eu-central-1
topology.kubernetes.io/zone=eu-central-1c

You could use beta.kubernetes.io/instance-type=t3.medium for example if it makes sense in your affinity rule, and you want to treat all nodes of instance-type=t3.medium as the same topology.

user_62426980
Related