I've run into some disk pressure issues under GKE (too many job pods that weren't cleaned up) and I want to somehow be alerted next time I run low on disk space. I looked in Stackdriver Monitoring, but I only see disk read/write bytes/operations. In order to get disk usage/utilization, I think I have to install the stackdriver agent, but I'm not sure if this is possible under GKE. Is there some way to monitor disk utilization for GKE nodes?
2 Answers
when installing the Stackdriver Agent, there are per default only these two disk metrics available: container/disk/bytes_total and container/disk/bytes_used. in fact, one cannot really monitor the disk utilization alike that.
rather see Creating Custom Metrics, with an Alerting Policy.
Was searching for this today. The recommended answer relates to a legacy metric. Now the following metric is available: metric.type="kubernetes.io/node/ephemeral_storage/allocatable_bytes" resource.type="k8s_node"
Related