Is there a way to view the history of pod Termination statuses? Eg. If I look at pod describe command I see output similar to this:
State: Running
Started: Mon, 10 Jul 2017 13:09:20 +0300
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Thu, 06 Jul 2017 11:01:21 +0300
Finished: Mon, 10 Jul 2017 13:09:18 +0300
The same pod describe does not show anything in the pod events:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
10m 10m 1 kubelet, gke-dev-default-d8f2dbc5-mbkb spec.containers{demo} Normal Pulled Container image "eu.gcr.io/project/image:v1" already present on machine
10m 10m 1 kubelet, gke-dev-default-d8f2dbc5-mbkb spec.containers{demo} Normal Created Created container with id 1d857caae77bdc43f0bc90fe045ed5050f85436479073b0e6b46454500f4eb5a
10m 10m 1 kubelet, gke-dev-default-d8f2dbc5-mbkb spec.containers{demo} Normal Started Started container with id 1d857caae77bdc43f0bc90fe045ed5050f85436479073b0e6b46454500f4eb5a
If I look at the kubectl get events --all-namespaces I see this event, but there is no way how to relate this to the particular pod:
default 12m 12m 1 gke-dev-default-d8f2dbc5-mbkb Node Warning OOMKilling kernel-monitor, gke-dev-default-d8f2dbc5-mbkb Memory cgroup out of memory: Kill process 1639 (java) score 2014 or sacrifice child
Killed process 1639 (java) total-vm:10828960kB, anon-rss:1013756kB, file-rss:22308kB
And even the event Details repoted through the api have missleading info (like the defaultnamespace though pod is actualy in demonamespace):
"metadata": {
"name": "gke-dev-default-d8f2dbc5-mbkb.14cff03fe771b053",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/events/gke-dev-default-d8f2dbc5-mbkb.14cff03fe771b053",
"uid": "d5d3230e-6557-11e7-a486-42010a8401d3",
"resourceVersion": "5278875",
"creationTimestamp": "2017-07-10T10:09:18Z"
},
"involvedObject": {
"kind": "Node",
"name": "gke-dev-default-d8f2dbc5-mbkb",
"uid": "gke-dev-default-d8f2dbc5-mbkb"
},
"reason": "OOMKilling",
"message": "Memory cgroup out of memory: Kill process 1639 (java) score 2014 or sacrifice child\nKilled process 1639 (java) total-vm:10828960kB, anon-rss:1013756kB, file-rss:22308kB",
"source": {
"component": "kernel-monitor",
"host": "gke-dev-default-d8f2dbc5-mbkb"
},
"firstTimestamp": "2017-07-10T10:09:18Z",
"lastTimestamp": "2017-07-10T10:09:18Z",
"count": 1,
"type": "Warning"
So while I can see last Termination status via pod describe, what about the previous ones?