I deployed a new app to GKE, I see the GKE dashboard has thousands of errors on gke-metrics-agent:
It uses a high resource.
I checked the logs, and I saw all errors related to Prometheus, but I didn't find a way to troubleshoot these errors:
cluster version: 1.18.12-gke.1206
What are these errors, and how I can fix it?
It looks like some GKE 1.18.12-gke-X versions have bug where gke-metrics-agent produces a lot of Warning messages.
There is already a Public Issue Tracker ticket for this bug. You can follow updates regarding this issue here. You can also use (+1) to indicate that you are affected by this bug.
Workaround of this issue is to use newer version - 1.18.14-gke.1200+
