Monitoring pod termination time with prometheus

user_3833773 0
Viewed 3635

I'm looking for a Prometheus metric that would allow me to monitor the time pods spend in the terminating state before vanishing into the void.

I've tried playing around with kube_pod_container_status_terminated but it only seems to register pods once they finish the termination process, but don't help me understand how long does it take to terminate a pod.
I've also looked at kube_pod_status_phase which I found out about in this channel a while ago but it also seems to lack this insight.

I'm currently collecting metrics on my k8s workload using cAdvisor, kube-state-metrics and the prometheus node-exporter, but would happily considering additional collectors if they contain the desired data.
A non-prometheus solution would also be great.
Any ideas? Thanks!

prometheus kubernetes
2 Answers

Kubernetes itself, Heapster and metrics-server don't provide such metrics, but you can get metrics close to what you've mentioned by installing kube-state-metrics. It has several pod metrics that reflect pods state:

kube_pod_status_phase
kube_pod_container_status_terminated
kube_pod_container_status_terminated_reason
kube_pod_container_status_last_terminated_reason

You can find the full list of pods metrics, provided by kube-state-metrics in Documentation.

There is also Bitnami Helm chart that could simplify the installation of kube-state-metrics.

user_95216100

As per pod-metrics documentation:

It is not straightforward to get the Pod states for certain cases like "Terminating" and "Unknown" since it is not stored behind a field in the Pod.Status.

So to mimic the logic used by the kubectl command line, you will need to compose multiple metrics. [...]

  • For Pods in Terminating state: count(kube_pod_deletion_timestamp) by (namespace, pod) * count(kube_pod_status_reason{reason="NodeLost"} == 0) by (namespace, pod)

Here is an example of a Prometheus rule that can be used to alert on a Pod that has been in the Terminated state for more than 5m.

groups:
- name: Pod state
  rules:
  - alert: PodsBlockInTerminatingState
    expr: count(kube_pod_deletion_timestamp) by (namespace, pod) * count(kube_pod_status_reason{reason="NodeLost"} == 0) by (namespace, pod) > 0
    for: 5m
    labels:
      severity: page
    annotations:
      summary: Pod {{$labels.namespace}}/{{$labels.pod}} block in Terminating state.
user_76410780
Related