"Predicate NodeAffinity failed"

user_16545833 0
Viewed 2174

I have a GKE cluster which I updated to "v1.19.11-gke.2101".

In that cluster we are running tons of pods which have node selectors like this:

      nodeSelector:
        cloud.google.com/gke-nodepool: default-pool

Since I updated the cluster, from time to time the pods are entering in:

Status:         Failed
Reason:         NodeAffinity"

With these status events:

  Warning  NodeNotReady  55m   node-controller                                            Node is not ready
  Warning  NodeAffinity  53m   kubelet, gke-ef-gke-cluster-front-default-pool-bbda0bbf-t4js  Predicate NodeAffinity failed

The thing is that even though the status is "fail", other pods are successfully scheduled so there is no "real issue" just an annoying red flag which I would like to get rid of.

I saw that there are several issues closed in the kubernets GitHub repo mentioning this and claiming that is fixed:

But the problem is still there in GKE 1.19.11. AFAIK gke has its own Kubernetes implementation so.. does any body know if it is actually solved in a specific GKE version? which one? And even if not.. Any idea of how to get rid of annoying flag without having to manually/periodically delete the stuck pods?

Thanks in advance, any help will be more than welcome.

google-kubernetes-engine
1 Answers

This issue is affecting GKE Clusters under the ‘1.19.11’ version, so upgrading them to ‘1.20.x’ would be a workaround if you are using preemptible Nodes. Please take into consideration that this issue is being currently worked on in order to provide a fixed solution.

Be aware that, although you may experience delay in the data processing, there should be no impact on the workload.

  • There is no alternate method for removing the failed pods other than the manual deletion, but they can re-appear as the issue has not been fixed.

  • Opting-out of preemptible nodes temporarily as this issue happens rarely on regular nodes.

You can track the updates in this new issue tracker

user_157451360
Related