We have GitLab cloud setup for CI/CD Pipelines using Gitlab runner on GKE Nodes. Where we have two node pools, One nodes pool is being used to host services from Gitlab. Second NodePool is being used for Gitlab runners.
In the First Node pool, we have 2 nodes; in the Second, we have an Autoscaling setup ( 0-10) based on the number of Gitlab Jobs running.
We are persistently facing the issue of Pipeline failure due to the Runner pod getting failed as can be seen in below log : ERROR: Job failed: pod "runner-qmyax7z-project-19482920-concurrent-7xtw8x" status is "Failed"
When searching in GCP Log Explorer logs related to the failed pod and the node on which it was scheduled found out that "Node Shutdown Manager" was triggering the Node Shutdown due to Node in "NodeNotReady" State :
Node gke-gitlab-gitlab-runners-a10e2804-822r status is now: NodeNotReady
nodeshutdown_manager_linux.go:224] "Shutdown manager processing shutdown event"
By describing the Node found out that in the "conditions" section kubelet is in "kubeletnotready" state and there is no issue related to scarcity of resources.
KUBELETNOTREADY Then checked in the kubelet logs from Logs Explorer and didn't found any suspicious log which would cause "kubelet" service to go down. It had only logs related to Pods getting started by kubelet as given below - kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-proxy-gke-gitlab-gitlab-runners-a10e2804-822r" status=Running
Can someone please advise on how to find the root cause of kubelet service restarting and getting in to "KubeletNotReady" State so that the Node Shutdown event can be resolved and GitLab pipeline jobs gets completed successfully.