CPUThrottlingHigh alert for metrics-server-nanny container in GKE

user_1637116 0
Viewed 2292

I noticed some of my clusters were reporting a CPUThrottlingHigh alert for metrics-server-nanny container (image: gke.gcr.io/addon-resizer:1.8.11-gke.0) in GKE. I couldn't see a way to configure this container to give it more CPU because it's automatically deployed as part of the metrics-server pod, and Google automatically resets any changes to the deployment/pod resource settings.

So out of curiosity, I created a small kubernetes cluster in GKE (3 standard nodes) with autoscaling turned on to scale up to 5 nodes. No apps or anything installed. Then I installed the kube-prometheus monitoring stack (https://github.com/prometheus-operator/kube-prometheus) which includes the CPUThrottlingHigh alert. Soon after installing the monitoring stack, this same alert popped up for this container. I don't see anything in the logs of this container or the related metrics-server-nanny container.

Also, I don't notice this same issue on AWS or Azure because while they do have a similar metrics-server pod in the kube-system namespace, they do not contain the sidecar metrics-server-nanny container in the pod.

Has anyone seen this or something similar? Is there a way to give this thing more resources without Google overwriting config changes?

autoscaling google-kubernetes-engine kubernetes google-cloud-platform monitoring
2 Answers

This is a known issue in Kubernetes that CFS leads to Throttling Pods that exhibit a spikey CPU usage pattern. As Kubernetes / GKE uses to implement CPU quotas, this is causing pods to get throttled even when they really aren't busy.

Kubernetes uses CFS quotas to enforce CPU limits for the pods running an application. The Completely Fair Scheduler (CFS) is a process scheduler that handles CPU resource allocation for executing processes, based on time period and not on available CPU power.

We have no direct control over CFS via Kubernetes, so the only solution is to disable it. This is done via node config.

Allow users to tune Kubelet configs "CPUManagerPolicy" and "CPUCFSQuota”

The workaround is to temporarily disable Kubernetes CFS quotas entirely (kubelet's flag --cpu-cfs-quota=false)

    $ cat node-config.yaml
    kubeletConfig:
    cpuCFSQuota: false
    cpuManagerPolicy: static

    $ gcloud container clusters create --node-config=node-config.yaml

gcloud will map the fields from the YAML node config file to the newly added GKE API fields.

user_157451360
user_4959950
Related