When I run shutdown -h now command to shutdown a node in kubernetes cluster, endpoint update its state after about 40 seconds, but when I run command kubectl delete pod POD-NAME, endpoint update its state very quick. Can anyone explain why?
Endpoint update very slow when shutdown a node
Viewed 133
2 Answers
When you "shutdown" a node, you should do it gracefully with kubectl drain. This will evict the pods in a controlled manner and this should be more friendly to your traffic.
The article Kubernetes best practices: terminating with grace has a detailed description on all steps that happen when a Pod is gracefully terminated. For all planned maintenance, use gracefully shutdown - for unplanned maintenance you can not do much.
I found the solution from here:
It may be caused by that the node is marked as NotReady after a grace period. This is configurable. After that, pod will be terminating and rescheduled
--node-monitor-grace-period duration Default: 40s
Amount of time which we allow running Node to be unresponsive before marking it unhealthy. Must be N times more than kubelet's nodeStatusUpdateFrequency, where N means number of retries allowed for kubelet to post node status.refer to: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
When I change the value of --node-monitor-grace-period to 10s, endpoint update its state more quick.
Related