GKE cluster suddenly not autoscaling nodepool

user_2568670 0
Viewed 1260

I have two nodepools on GKE for CI runners, the workers nodepool is set to autoscale between 0-40 nodes. Up until yesterday this has been working perfectly for ~6months but now I am getting an inconsistent error around nap being disabled.

Pods scheduling fine this morning:

decision: {
  decideTime: "1616746520" // Friday, 26 March 2021 08:15:20 GMT
  scaleUp: {

At some point they started failing because of nap.disabled:

noDecisionStatus: {
  measureTime: "1616756707" // Friday, 26 March 2021 11:05:07 GMT
  napFailureReason: {
    messageId: "no.scale.up.nap.disabled"
  }

This is technically correct in that the cluster itself has node auto-provisioning disabled, but the nodepool workers2 has auto scaling from 0-40 nodes.

I can't seem to find any information around why I would get this error when the nodepool itself should scale, nor can I see any reason why pods would suddenly not trigger the autoscaling of the nodepool.

Any ideas where I can dig to find more info or stop that error from blocking the autoscaling?

google-kubernetes-engine kubernetes
2 Answers

Creating a new node pool and destroying the old one seems to have fixed this, unfortunately no more insight than that

user_25686700
user_68219360
Related