Get Pods on node, sorted by disk space usage

Viewed 540

Does anyone know how to get a sorted list of pods on a given node based on their disk space consumption?

The command below helps me in listing pods based on a given node but my requirement is to list those pods which are causing high disk space usage as part of investigating and solving the DiskPressure eviction state.

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<NODE_NAME>

I was able to find commands to list CPU and memory usage of nodes (ref: https://github.com/kubernetes/kubernetes/issues/17512#issuecomment-317757388) but nothing related to node disk usage.

alias util='kubectl get nodes --no-headers | awk '\''{print $1}'\'' | xargs -I {} sh -c '\''echo {} ; kubectl describe node {} | grep Allocated -A 5 | grep -ve Event -ve Allocated -ve percent -ve -- ; echo '\'''

# Get CPU request total (we x20 because because each m3.large has 2 vcpus (2000m) )
alias cpualloc='util | grep % | awk '\''{print $1}'\'' | awk '\''{ sum += $1 } END { if (NR > 0) { print sum/(NR*20), "%\n" } }'\'''

# Get mem request total (we multiply by 75 because because each m3.large has 7.5G ram )
alias memalloc='util | grep % | awk '\''{print $5}'\'' | awk '\''{ sum += $1 } END { if (NR > 0) { print sum/(NR*75), "%\n" } }'\'''

2 Answers

found an answer to my requirement - list pods disk usage running in a given node

kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.pods[0] | "PodName: ", .podRef.name, "usedBytes: ", .containers[].rootfs.usedBytes'

Some other stats:

get node filesystem usage

kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.fs.usedBytes'

get node imageFs usage

kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.runtime.imageFs.usedBytes'

get node iNodes stats

kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.fs.inodesFree'

kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.runtime.imageFs.inodesFree'

Unfortunately you won't be able to achieve this with kubectl as it does not have this kind of functionality. One way to go is login to node and the use docker ps -s.

Another way that I can think of is to expose kubelet/cadvisor metrics and use it with metrics scraper like Prometheus.

The kubelet exposes all of it’s runtime metrics, and all of the cAdvisor metrics, on a /metrics endpoint in the Prometheus exposition format. Note that kubelet also exposes metrics in /metrics/cadvisor, /metrics/resource and /metrics/probes endpoints. Disk metrics are generally available on the /stats/summary endpoint.

Additional info:

Related