How to get container's disk usage in Kubernetes (without docker command)?

user_5902456 0
Viewed 11930

I have a Kubernetes Cluster and want to know how much disk space my containers use. I am not talking about mounted Volumes.

I can get this information by using docker commands like docker system df -v or docker ps -s, but I don't want to connect to every single worker node.

Is there a way to get a container's disk usage via kubectl or are there kubelet metrics where I can get this information from?

kubectl prometheus kubernetes
3 Answers

Yes, but currently not with kubectl, you can get metrics from the kubelet, either through the kube-apiserver (proxied) or directly calling the kubelet HTTP(s) server endpoint (default port 10250). Disk metrics are generally available on the /stats/summary endpoint and you can also find some cAdvisor metrics on the /metrics/cavisor endpoint.

For example, to get the 'usedBytes' for the first container in the first pod returned through the kube-apiserver:

$ curl -k -s -H 'Authorization: Bearer <REDACTED>' \
  https://kube-apiserver:6443/api/v1/nodes/<node-name>/proxy/stats/summary \
  | jq '.pods[0].containers[0].rootfs.usedBytes'

The Bearer token can be a service account token tied to a ClusterRole like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
  name: myrole
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  - /api/*
  verbs:
  - get
user_29892610

Adding to Rico's answer you can use kubectl to set up a proxy to the API server so that you can query the API with the details of the server address and authentication taken care of.

Set up a local proxy to the API server with

kubectl proxy --port=8080

Then you can use a command like

curl http://localhost:8080/api/v1/nodes/<node-name>/proxy/stats/summary 2>/dev/null | jq '.pods[].containers[].rootfs.usedBytes'

which will give you a list of the rootfs usage by each container.

user_2177540

To add on to Stephen's answer, here is a curl query if you want the pod name alongside the size in bytes.

curl http://localhost:8080/api/v1/nodes/<node-name>/proxy/stats/summary 2>/dev/null | jq '.pods[].containers[] | "\(.name) \(.rootfs.usedBytes)"'
user_90360560
Related