How can I check the time taken by k8s pod to start?

user_10805230 0
Viewed 3883

I want to know the time it takes to start my pod in Kubernetes. My pod has 3 Init Containers and 9 actual Containers. Here's what I have tried so far:

  • Approach 1: Start the pod and monitor until there are 9/9 containers running. The AGE in kubectl describe pod <pod_name> will give an idea for startup time.
  • Approach 2: Refer to the events for the pod using kubectl events or kubectl describe command. The problem is events are not there for all containers. Also, there is no clear way to understand the time taken by pod to be in READY state.

enter image description here

Question: Does Kubernetes provide any way to determine the whole pod startup time?

kubernetes-pod kubernetes
3 Answers

As things begin to set up, your pod will progress through a set of PodConditions.

You could use:

kubectl get po <pod_name> -o yaml

to observe the progression through these various conditions:

...
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-09-09T08:47:11Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-09-09T08:47:12Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-09-09T08:47:12Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-09-09T08:47:11Z"
    status: "True"
    type: PodScheduled
...

If you want to automatically parse all of this in a shell, I'd recommend yq for the actual YAML parsing.

user_63829010

While there is no exact mechanism for that in Kubernetes you may want to check Kube-state-metrics:

kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. (See examples in the Metrics section below.) It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.

With it you can export object creation time. The metric name in that case would be kube_<OBJECT>_created. This article about metrics in section 6 explain more about kube state metrics and usage of object creation time.

Note that if you are about to measure your pod start time you have assume that all images needed to run that pod are already pre-pulled on the machine. Otherwise your measurement will not be correct as it will include variables that influence Kubernetes performance such as network or image size.

user_121865850

In case this can help anyone out there, here is an example using kubectl, yq and bash

$ kubectl -n ${NAMESPACE} get pods -o yaml | yq e '.items[].status.conditions[] | select('.type' == "PodScheduled" or '.type' == "Ready") | '.lastTransitionTime'' - | xargs -n 2 bash -c 'echo $(( $(date -d "$0" "+%s") - $(date -d "$1" "+%s") ))'
67
70
70
72
...
user_5838820
Related