Scheduler doesn't respect topologySpreadConstraints

user_10071806 0
Viewed 520

I have a nodegroup on eks with 2 nodes, and a deployment with 6 replicas. I'm trying to have like 3 pods per node but it never get spread evenly. I have a taint on that node group and toleration in the deployment.

      tolerations:
      - effect: NoSchedule
        key: type
        operator: Equal
        value: ram
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app: service-vehicles
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule

Sometimes it's 4 pods on node 1 and 2 pods on node 2 I have another deployment with 3 replicas, it puts the 3 pods on one node.

kubectl describe on these nodes: node1:

Name:               ip-192-168-189-82.eu-west-1.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=m5a.xlarge
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/capacityType=ON_DEMAND
                    eks.amazonaws.com/nodegroup=ram
                    eks.amazonaws.com/nodegroup-image=ami-08adae443f6a2b4f6
                    failure-domain.beta.kubernetes.io/region=eu-west-1
                    failure-domain.beta.kubernetes.io/zone=eu-west-1c
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-192-168-189-82.eu-west-1.compute.internal
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=m5a.xlarge
                    topology.kubernetes.io/region=eu-west-1
                    topology.kubernetes.io/zone=eu-west-1c
                    type=ram
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 19 Jun 2021 13:59:51 +0200
Taints:             type=ram:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  ip-192-168-189-82.eu-west-1.compute.internal
  AcquireTime:     <unset>
  RenewTime:       Sat, 19 Jun 2021 18:51:13 +0200
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Sat, 19 Jun 2021 18:50:57 +0200   Sat, 19 Jun 2021 13:59:51 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sat, 19 Jun 2021 18:50:57 +0200   Sat, 19 Jun 2021 13:59:51 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sat, 19 Jun 2021 18:50:57 +0200   Sat, 19 Jun 2021 13:59:51 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Sat, 19 Jun 2021 18:50:57 +0200   Sat, 19 Jun 2021 14:00:11 +0200   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:   192.168.189.82
  Hostname:     ip-192-168-189-82.eu-west-1.compute.internal
  InternalDNS:  ip-192-168-189-82.eu-west-1.compute.internal
Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         4
  ephemeral-storage:           31444972Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      16009732Ki
  pods:                        58
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         3920m
  ephemeral-storage:           27905944324
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      14992900Ki
  pods:                        58
System Info:
  Machine ID:                 ec2c02c406fcdee288eec2474047aad2
  System UUID:                ec2c02c4-06fc-dee2-88ee-c2474047aad2
  Boot ID:                    2e9fa901-a603-472e-8dd8-d5078a67e1b2
  Kernel Version:             5.4.117-58.216.amzn2.x86_64
  OS Image:                   Amazon Linux 2
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.13
  Kubelet Version:            v1.19.6-eks-49a6c0
  Kube-Proxy Version:         v1.19.6-eks-49a6c0
ProviderID:                   aws:///eu-west-1c/i-0c5b8bfe55c93f684
Non-terminated Pods:          (13 in total)
  Namespace                   Name                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                         ------------  ----------  ---------------  -------------  ---
  cert-manager                cert-manager-webhook-865598c587-ljfnz        0 (0%)        0 (0%)      0 (0%)           0 (0%)         4h30m
  ingress-nginx               ingress-nginx-controller-6db8d48bd8-479tt    100m (2%)     0 (0%)      90Mi (0%)        0 (0%)         4h30m
  ingress-nginx               ingress-nginx-controller-6db8d48bd8-6f7fr    100m (2%)     0 (0%)      90Mi (0%)        0 (0%)         4h30m
  kube-system                 aws-node-lgj69                               10m (0%)      0 (0%)      0 (0%)           0 (0%)         4h51m
  kube-system                 aws-node-termination-handler-26jp7           50m (1%)      100m (2%)   64Mi (0%)        128Mi (0%)     4h51m
  kube-system                 coredns-c547bd959-hx6xq                      100m (2%)     0 (0%)      70Mi (0%)        170Mi (1%)     4h30m
  kube-system                 kube-proxy-cjkts                             100m (2%)     0 (0%)      0 (0%)           0 (0%)         4h51m
  kube-system                 metrics-server-74cb5875d8-7vl2m              0 (0%)        0 (0%)      0 (0%)           0 (0%)         4h30m
  kubernetes-dashboard        dashboard-metrics-scraper-dc7b7d4c4-nfkbp    0 (0%)        0 (0%)      0 (0%)           0 (0%)         4h30m
  kubernetes-dashboard        kubernetes-dashboard-68cf866dbf-9vfqt        0 (0%)        0 (0%)      0 (0%)           0 (0%)         4h30m
  production                  cars-ai-74b86f79c7-2xdm5              200m (5%)     350m (8%)   800Mi (5%)       1600Mi (10%)   4h12m
  production                  cars-ai-74b86f79c7-t984b              200m (5%)     350m (8%)   800Mi (5%)       1600Mi (10%)   4h11m
  prometheus                  prometheus2-node-exporter-tbw4c              0 (0%)        0 (0%)      0 (0%)           0 (0%)         4h51m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         860m (21%)    800m (20%)
  memory                      1914Mi (13%)  3498Mi (23%)
  ephemeral-storage           0 (0%)        0 (0%)
  hugepages-1Gi               0 (0%)        0 (0%)
  hugepages-2Mi               0 (0%)        0 (0%)
  attachable-volumes-aws-ebs  0             0
Events:                       <none>

node2:

Name:               ip-192-168-73-159.eu-west-1.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=m5a.xlarge
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/capacityType=ON_DEMAND
                    eks.amazonaws.com/nodegroup=ram
                    eks.amazonaws.com/nodegroup-image=ami-08adae443f6a2b4f6
                    failure-domain.beta.kubernetes.io/region=eu-west-1
                    failure-domain.beta.kubernetes.io/zone=eu-west-1c
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-192-168-73-159.eu-west-1.compute.internal
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=m5a.xlarge
                    topology.kubernetes.io/region=eu-west-1
                    topology.kubernetes.io/zone=eu-west-1c
                    type=ram
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 18 Jun 2021 18:32:00 +0200
Taints:             type=ram:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  ip-192-168-73-159.eu-west-1.compute.internal
  AcquireTime:     <unset>
  RenewTime:       Sat, 19 Jun 2021 18:58:19 +0200
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Sat, 19 Jun 2021 18:54:09 +0200   Sat, 19 Jun 2021 08:33:48 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sat, 19 Jun 2021 18:54:09 +0200   Sat, 19 Jun 2021 08:33:48 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sat, 19 Jun 2021 18:54:09 +0200   Sat, 19 Jun 2021 08:33:48 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Sat, 19 Jun 2021 18:54:09 +0200   Sat, 19 Jun 2021 08:33:48 +0200   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:   192.168.73.159
  ExternalIP:   52.209.118.101
  Hostname:     ip-192-168-73-159.eu-west-1.compute.internal
  InternalDNS:  ip-192-168-73-159.eu-west-1.compute.internal
  ExternalDNS:  ec2-52-209-118-101.eu-west-1.compute.amazonaws.com
Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         4
  ephemeral-storage:           31444972Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      16009732Ki
  pods:                        58
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         3920m
  ephemeral-storage:           27905944324
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      14992900Ki
  pods:                        58
System Info:
  Machine ID:                 ec29049c0c24c40d2951cfc7ca1c39cd
  System UUID:                ec29049c-0c24-c40d-2951-cfc7ca1c39cd
  Boot ID:                    fe117d3d-e744-4143-9be3-fadc3435c6f0
  Kernel Version:             5.4.117-58.216.amzn2.x86_64
  OS Image:                   Amazon Linux 2
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.13
  Kubelet Version:            v1.19.6-eks-49a6c0
  Kube-Proxy Version:         v1.19.6-eks-49a6c0
ProviderID:                   aws:///eu-west-1c/i-0b783ab02564f267f
Non-terminated Pods:          (14 in total)
  Namespace                   Name                                          CPU Requests  CPU Limits   Memory Requests  Memory Limits  Age
  ---------                   ----                                          ------------  ----------   ---------------  -------------  ---
  grafana                     grafana-6dd545dbd8-lj8zz                      0 (0%)        0 (0%)       0 (0%)           0 (0%)         24h
  kube-system                 aws-node-qn8fh                                10m (0%)      0 (0%)       0 (0%)           0 (0%)         24h
  kube-system                 aws-node-termination-handler-vzhfn            50m (1%)      100m (2%)    64Mi (0%)        128Mi (0%)     24h
  kube-system                 kube-proxy-f5pgr                              100m (2%)     0 (0%)       0 (0%)           0 (0%)         24h
  production                  cars-centers-8b565749-dt76m            60m (1%)      100m (2%)    80Mi (0%)        300Mi (2%)     24h
  production                  cars-cms-5484bfc78c-vrzps              60m (1%)      100m (2%)    80Mi (0%)        300Mi (2%)     24h
  production                  cars-cms-5484bfc78c-xxdjt              60m (1%)      100m (2%)    80Mi (0%)        300Mi (2%)     24h
  production                  cars-notifications-7c9b76544f-fv9v9    1600m (40%)   2400m (61%)  3500Mi (23%)     4000Mi (27%)   9h
  production                  cars-portal-7c9bff785-ngff7            800m (20%)    900m (22%)   3000Mi (20%)     4000Mi (27%)   4h49m
  production                  cars-vehicles-59474d89c6-qljf8         300m (7%)     350m (8%)    1000Mi (6%)      1300Mi (8%)    4h59m
  production                  cars-vehicles-59474d89c6-t4dg4         300m (7%)     350m (8%)    1000Mi (6%)      1300Mi (8%)    4h59m
  prometheus                  prometheus2-alertmanager-5b5bb9bc49-g5pbx     0 (0%)        0 (0%)       0 (0%)           0 (0%)         24h
  prometheus                  prometheus2-node-exporter-f75nz               0 (0%)        0 (0%)       0 (0%)           0 (0%)         24h
  prometheus                  prometheus2-server-74df6cbc7b-bbbp8           0 (0%)        0 (0%)       0 (0%)           0 (0%)         24h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         3340m (85%)   4400m (112%)
  memory                      8804Mi (60%)  11628Mi (79%)
  ephemeral-storage           0 (0%)        0 (0%)
  hugepages-1Gi               0 (0%)        0 (0%)
  hugepages-2Mi               0 (0%)        0 (0%)
  attachable-volumes-aws-ebs  0             0
Events:                       <none>
amazon-eks kubernetes scheduler
1 Answers

I think you need to mention maxSkew there. So maxSkew means how much difference there can be in pod placements among nodes. So if you mention maxSkew as 1 and you have 3 pods and 2 eks nodes, then one of the node will have 2 pods ad other will have 1 so 2-1=1, hence maxSkew is 1. In your case it should be 0 i think if you have even number of pods to be placed on all nodes. However the documentation says maxskew should be greater than zero. So you are left with 1. Just an idea https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#spread-constraint-definition

          topologySpreadConstraints:
            - maxSkew: 1
              topologyKey: topology.kubernetes.io/zone
              whenUnsatisfiable: DoNotSchedule
              labelSelector:
                matchLabels:
                  elasticsearch.k8s.elastic.co/cluster-name: my_cluster

One thing you could do is to limit your node to have only certain number of pods which matches a label. I havent done it myself but i think the documentation might help you in this regard https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity

user_54908630
Related