This mainly for GKE cluster with preemptible nodes. Google Cloud would shutdown any node without warning, which leaves lots of pods with Terminated and / or NodeShutdown state. My current UAT environment is a GKE cluster with hundreds of tenants, where each tenant occupies one namespace. Delete all pods with Terminated and / or NodeShutdown state (means across namespace) would take a very time and not really practical as single run, so I think it might be better for each namespace running its own 'janitor' Cronjob. Now, I could create Cronjob manually with existing namespaces, but I can't stop my colleagues to create other namespaces for their tenants. So I start to think of if I could do similar thing just like Istio auto sidecar injection.
Just curious anybody know what documents / github repo I should dig from here. Or, if what I thought might be totally wrong, please point a correct way to solve this problem.