I'm working on a testing system that uses Kubernetes (K8S) as the testing infrastructure. The general model of our system is that a scheduling system takes a docker image and creates many jobs on K8S - one per test. Each job is configured such that only one pod should run. The scheduler uses watches to monitor the K8S namespace for all Job and Pod events.
This mostly works fine - events come in and are processed. However, our monitoring showed that a number of jobs were staying in the "Completed" state for many minutes (15+) - as though the stream of events for that job/pod had dropped events. This has eventually lead to various band aids - we now restart the watchers every 30s, and additionally have a polling loop that queries K8S for jobs that have succeeded.
We are keeping comprehensive logs of all events that we process, and for the jobs in question there are no events to indicate that state has changed - yet requesting the job/pod info shows an updated state.
The job watcher is configured as
timeout := int64(30)
return k.clientset.BatchV1().Jobs(namespace).Watch(context.TODO(), metav1.ListOptions{TimeoutSeconds: &timeout})
and the pod watch is identical. The monitoring loop around these calls looks like
func monitorAllPods() {
for {
watch, err := kube.podWatch()
check(err)
for event := range watch.ResultChan() {
pod, ok := event.Object.(*corev1.Pod)
if !ok {
log.Printf("Got event %v, but couldn't convert to pod\n", event)
continue
}
handlePodEvent(event.Type, pod)
}
}
}
Am I missing something obvious? Are others seeing this sort of strangeness?
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}