How to get CPU and memory usage of pod in percentage using promethus

user_3956228 0
Viewed 9460

I want to display pod details in the following format using promql/Prometheus.

Image1

Furthermore, I want to display CPU and memory utilization of application/component in below format using promql

Image2

promql query: sum(container_memory_working_set_bytes) by (pod)

I can get the consumed memory by pod using above query.

How to calculate percentage of memory used ? I am not able to fetch memory limit of stateful pod using promql Could you please suggest any query/API details ?

grafana prometheus promql kubernetes monitoring
3 Answers

Per-pod CPU usage in percentage (the query doesn't return CPU usage for pods without CPU limits)

100 * max(
  rate(container_cpu_usage_seconds_total[5m])
    / on (container, pod)
  kube_pod_container_resource_limits{resource="cpu"}
) by (pod)

The kube_pod_container_resource_limits metric can be scraped incorrectly if scrape config for kube-state-metrics pod is improperly configured. In this case the original pod label for this metric is moved to the exported_pod label because of honor_labels behavior - see these docs for details. In this case label_replace function must be used for moving exported_pod label to pod label:

100 * max(
  rate(container_cpu_usage_seconds_total[5m])
    / on (container, pod)
  label_replace(kube_pod_container_resource_limits{resource="cpu"}, "pod", "$1", "exported_pod", "(.+)")
) by (pod)

Per-pod memory usage in percentage (the query doesn't return memory usage for pods without memory limits)

100 * max(
  container_memory_working_set_bytes
    / on (container, pod)
  kube_pod_container_resource_limits{resource="memory"}
) by (pod)

If the kube_pod_container_resource_limits metric is scraped incorrectly as mentioned above, then the label_replace function must be used for moving exported_pod label value to pod:

100 * max(
  container_memory_working_set_bytes
    / on (container, pod)
  label_replace(kube_pod_container_resource_limits{resource="memory"}, "pod", "$1", "exported_pod", "(.+)")
) by (pod)
user_2749370

For CPU percentage

max by (pod) (sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",container_name!="POD",container_name!="",container!="monitoring-daemon"}[5m])) / sum(kube_pod_container_resource_limits{namespace="$namespace", resource="cpu"})) * 100

For Memory percentage

avg((avg (container_memory_working_set_bytes{pod="<Podname>"}) by (container_name , pod ))/ on (container_name , pod)(avg (container_spec_memory_limit_bytes>0 ) by (container_name, pod))*100)
user_39562280

I use this for memory:

round(max by (pod)(max_over_time(container_memory_usage_bytes{namespace="$namespace",pod=~".*" }[5m]))/ on (pod) (max by (pod) (kube_pod_container_resource_limits)) * 100,0.01)

And this for CPU :

max by (pod) (sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",container_name!="POD",container_name!="",container!="monitoring-daemon"}[5m])) / sum(kube_pod_container_resource_limits{namespace="$namespace", resource="cpu"})) * 100
user_70469600
Related