'Kubelet stopped posting node status' and node inaccessible

user_3410112 0
Viewed 3816

I am having some issues with a fairly new cluster where a couple of nodes (always seems to happen in pairs but potentially just a coincidence) will become NotReady and a kubectl describe will say that the Kubelet stopped posting node status for memory, disk, PID and ready.

All of the running pods are stuck in Terminating (can use k9s to connect to the cluster and see this) and the only solution I have found is to cordon and drain the nodes. After a few hours they seem to be being deleted and new ones created. Alternatively I can delete them using kubectl.

They are completely inaccessible via ssh (timeout) but AWS reports the EC2 instances as having no issues.

This has now happened three times in the past week. Everything does recover fine but there is clearly some issue and I would like to get to the bottom of it.

How would I go about finding out what has gone on if I cannot get onto the boxes at all? (Actually just occurred to me to maybe take a snapshot of the volume and mount it so will try that if it happens again, but any other suggestions welcome)

Running kubernetes v1.18.8

kubernetes amazon-web-services
3 Answers

There are two most common possibilities here, both most likely caused by a large load:

  • Out of Memory error on the kubelet host. Can be solved by adding proper --kubelet-extra-args to BootstrapArguments. For example: --kubelet-extra-args "--kube-reserved memory=0.3Gi,ephemeral-storage=1Gi --system-reserved memory=0.2Gi,ephemeral-storage=1Gi --eviction-hard memory.available<200Mi,nodefs.available<10%"

  • An issue explained here:

kubelet cannot patch its node status sometimes, ’cos more than 250 resources stay on the node, kubelet cannot watch more than 250 streams with kube-apiserver at the same time. So, I just adjust kube-apiserver --http2-max-streams-per-connection to 1000 to relieve the pain.

You can either adjust the values provided above or try to find the cause of high load/iops and try to tune it down.

user_115608780

I had the same issue, after 20-30 min my nodes became in NotRready status, and all pods linked to these nodes became stuck in Terminating status.
I tried to connect to my nodes via SSH, sometimes I faced a timeout, sometimes I could (hardly) connect, and I executed the top command to check the running processes.
The most consuming process was kswapd0.
My instance memory and CPU were both full (!), because it tried to swap a lot (due to a lack of memory), causing the kswapd0 process to consume more than 50% of the CPU!

Root cause:
Some pods consumed 400% of their memory request (defined in Kubernetes deployment), because they were initially under-provisioned. As a consequence, when my nodes started, Kubernetes placed them on nodes with only 32Mb of memory request per pod (the value I had defined), but that was insufficient.

Solution:
The solution was to increase containers requests:

requests:
  memory: "32Mi"
    cpu: "20m"
    limits:
       memory: "256Mi"
       cpu: "100m"

With these values (in my case):

requests:
  memory: "256Mi"
    cpu: "20m"
    limits:
       memory: "512Mi"
       cpu: "200m"

Important:
After that I processed a rolling update (cordon > drain > delete) of my nodes in order to ensure that Kubernetes reserve directly enough memory for my freshly started pods.

Conclusion:
Regularly check your pods' memory consumption, and adjust your resources requests over time.
The goal is to never leave your nodes be surprised by a memory saturation, because the swap can be fatal for your nodes.

user_20180430

The answer turned out to be an issue with iops as a result of du commands coming from - I think - cadvisor. I have moved to io1 boxes and have had stability since then so going to mark this as closed and the move of ec2 instance types as the resolution

Thanks for the help!

user_34101120
Related