Append timestamp to kubernetes --watch-only command

user_1342281 0
Viewed 2310

I'm running a Kubernetes job, where I want to monitor the state. I'm running various --watch-only commands simultaneously, e.g kubectl get pods --watch-only, which shows me the updated state of pods. But, I want to have timestamp and some string appended to the output.

The idea is to know when the state changed and also add additional info as a string.

How can I achieve this?

kubectl kubernetes
2 Answers

Giving more visibility to the potential solution for the question posted in the comments by original poster:

This is what i found working so far kubectl get pods --watch-only | while read line ; do echo -e "$(date +"%Y-%m-%d %H:%M:%S.%3N")\t pods\t $line" ; done

Command used:

  • $ kubectl get pods --watch-only | while read line ; do echo -e "$(date +"%Y-%m-%d %H:%M:%S.%3N")\t pods\t $line" ; done

Solution is correct but the changes in state (PENDING,RUNNING,SUCCEEDED/COMPLETED) would need to be further extracted (assuming further actions) from the above command.

Looking on this from a different perspective you can use the official Kubernetes API library to monitor the statuses of pods and jobs and act accordingly to them (for example: do something when job succeeded).

I've created an example app with Kubernetes Python API library to watch the statuses of pods and jobs.

Assuming that:

  • You have a working Kubernetes cluster with kubectl configured
  • You have installed Python and required library:
    • $ pip install kubernetes

Using the example for a Job:

Kubernetes.io: Docs: Concepts: Job

Example for pods

Below is the sample code in Python3 that would watch for pods and print a message when status of pod is set to Succeeded:

from kubernetes import client, config, watch
from datetime import datetime

config.load_kube_config()

v1 = client.CoreV1Api()
w = watch.Watch()
for event in w.stream(v1.list_namespaced_pod, namespace="default", watch=False):
    print("POD_NAME: " + event['object'].metadata.name) # print the name
    print("TIME: " + str(datetime.now())) # print the time 
    print("PHASE: " +  event['object'].status.phase) # print the status of the pod
    print("CUSTOM AMAZING TEXT HERE!")

    if (event['object'].status.phase == "Succeeded") and (event['type'] != "DELETED"): # do below when condition is met 
        print ("----> This pod succeeded, do something here!")
    print("---")

This would produce an output similar to one below:

POD_NAME: pi-pjmm5
TIME: 2020-09-06 15:28:01.541244
PHASE: Pending
CUSTOM AMAZING TEXT HERE!
---
POD_NAME: pi-pjmm5
TIME: 2020-09-06 15:28:03.894063
PHASE: Running
CUSTOM AMAZING TEXT HERE!
---
POD_NAME: pi-pjmm5
TIME: 2020-09-06 15:28:09.044219
PHASE: Succeeded
CUSTOM AMAZING TEXT HERE!
----> This pod succeeded, do something here!
---

Example for jobs

Below is the sample code in Python3 that would watch for jobs and print a message when status of job is set to Succeeded:

from kubernetes import client, config, watch
from datetime import datetime

config.load_kube_config()

v1 = client.BatchV1Api()
w = watch.Watch()
for event in w.stream(v1.list_namespaced_job, namespace="default", watch=False):
    print("JOB_NAME: " + event['object'].metadata.name)
    print("TIME: " + str(datetime.now()))
    print("STATUS: " + event['type'])
    print("CUSTOM AMAZING TEXT HERE!")

    if (event['object'].status.succeeded == 1) and (event['type'] != "DELETED"): 
        print ("----> This job succeeded, do something here!")
    print("---")

This would produce an output similar to one below:

JOB_NAME: pi
TIME: 2020-09-06 15:32:49.909096
STATUS: ADDED
CUSTOM AMAZING TEXT HERE!
---
JOB_NAME: pi
TIME: 2020-09-06 15:32:49.936856
STATUS: MODIFIED
CUSTOM AMAZING TEXT HERE!
---
JOB_NAME: pi
TIME: 2020-09-06 15:32:56.998511
STATUS: MODIFIED
CUSTOM AMAZING TEXT HERE!
----> This job succeeded, do something here!
---
user_122571340

There is --timestamps for that:

kubectl logs -f --timestamps ...
user_54064990
Related