I'm running an azure kubernetes service with 1 system pool (agentpool) and 1 user pool (userpool).
The agentpool has a CriticalAddonsOnly=true:NoSchedule taint to avoid being used by non critical pods and is configured for autoscaling from 1 to 5 nodes.
The userpool is configured for autoscaling from 0 to 5 nodes.
I'm trying to test the autoscaling by running a lot of pods on the user node in the hope that the autoscaler kicks in and starts a second userpool node.
Looking at the events, I'm seeing pods being evicted due to a lack of resources like the following:
Evicted pod/runner-ypq5fqbh-project-635-concurrent-594whk The node was low on resource: memory. Container helper was using 10272Ki, which exceeds its request of 0. Container svc-0 was using 30116Ki, which exceeds its request of 0. Container build was using 1681384Ki, which exceeds its request of 0.
and
EvictionThresholdMet node/aks-userpool-27234786-vmss000004 Attempting to reclaim memory
I tried to keep scheduling pods for several minutes. In the meantime I watched the cluster-autoscaler-status and could not find any evidence of a second node being started:
kubectl describe configmap --namespace kube-system cluster-autoscaler-status
Name: cluster-autoscaler-status
Namespace: kube-system
Labels: <none>
Annotations: cluster-autoscaler.kubernetes.io/last-updated: 2022-05-18 14:30:15.943110841 +0000 UTC
Data
====
status:
----
Cluster-autoscaler status at 2022-05-18 14:30:15.943110841 +0000 UTC:
Cluster-wide:
Health: Healthy (ready=2 unready=0 notStarted=0 longNotStarted=0 registered=2 longUnregistered=0)
LastProbeTime: 2022-05-18 14:30:15.934929647 +0000 UTC m=+2120.420060352
LastTransitionTime: 2022-05-18 14:10:45.260804967 +0000 UTC m=+949.745935672
ScaleUp: NoActivity (ready=2 registered=2)
LastProbeTime: 2022-05-18 14:30:15.934929647 +0000 UTC m=+2120.420060352
LastTransitionTime: 2022-05-18 14:21:04.140370827 +0000 UTC m=+1568.625501532
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2022-05-18 14:30:15.934929647 +0000 UTC m=+2120.420060352
LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
NodeGroups:
Name: aks-agentpool-27234786-vmss
Health: Healthy (ready=1 unready=0 notStarted=0 longNotStarted=0 registered=1 longUnregistered=0 cloudProviderTarget=1 (minSize=1, maxSize=5))
LastProbeTime: 2022-05-18 14:30:15.934929647 +0000 UTC m=+2120.420060352
LastTransitionTime: 2022-05-18 14:10:45.260804967 +0000 UTC m=+949.745935672
ScaleUp: NoActivity (ready=1 cloudProviderTarget=1)
LastProbeTime: 2022-05-18 14:30:15.934929647 +0000 UTC m=+2120.420060352
LastTransitionTime: 2022-05-18 14:10:45.260804967 +0000 UTC m=+949.745935672
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2022-05-18 14:30:15.934929647 +0000 UTC m=+2120.420060352
LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
Name: aks-userpool-27234786-vmss
Health: Healthy (ready=1 unready=0 notStarted=0 longNotStarted=0 registered=1 longUnregistered=0 cloudProviderTarget=1 (minSize=0, maxSize=5))
LastProbeTime: 2022-05-18 14:30:15.934929647 +0000 UTC m=+2120.420060352
LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
ScaleUp: NoActivity (ready=1 cloudProviderTarget=1)
LastProbeTime: 2022-05-18 14:30:15.934929647 +0000 UTC m=+2120.420060352
LastTransitionTime: 2022-05-18 14:21:04.140370827 +0000 UTC m=+1568.625501532
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2022-05-18 14:30:15.934929647 +0000 UTC m=+2120.420060352
LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
BinaryData
====
Events: <none>
Any idea's why this does not work?
I'm using Kubernetes version 1.22.6 using Standard_D2as_v5 for both user and agent pools.