Pod stays in pending state due to failed scheduling

user_558569 0
Viewed 1971

I'm new to Kubernetes and I'm trying to get a deploy running.

After I pushed the deploy config the replica set is created and that one will create the pod. But the pod stays in the Pending state.

The pod has an event listed that it can't be scheduled because there are no nodes available. Output kubectl describe pod foo-qa-1616599440:

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  25m    default-scheduler  0/6 nodes are available: 6 Insufficient pods.
  Warning  FailedScheduling  18m    default-scheduler  0/6 nodes are available: 6 Insufficient pods.
  Warning  FailedScheduling  11m    default-scheduler  0/6 nodes are available: 6 Insufficient pods.
  Warning  FailedScheduling  5m18s  default-scheduler  0/6 nodes are available: 6 Insufficient pods.

But there are Nodes available. Output of kubectl get nodes:

NAME                                             STATUS   ROLES    AGE   VERSION
ip-xxx-xx-xx-xx.eu-central-1.compute.internal   Ready    <none>   64d   v1.17.12-eks-7684af
ip-xxx-xx-xx-xx.eu-central-1.compute.internal   Ready    <none>   64d   v1.17.12-eks-7684af
ip-xxx-xx-xx-xx.eu-central-1.compute.internal   Ready    <none>   54d   v1.17.12-eks-7684af
ip-xxx-xx-xx-xx.eu-central-1.compute.internal   Ready    <none>   64d   v1.17.12-eks-7684af
ip-xxx-xx-xx-xx.eu-central-1.compute.internal   Ready    <none>   54d   v1.17.12-eks-7684af
ip-xxx-xx-xx-xx.eu-central-1.compute.internal   Ready    <none>   64d   v1.17.12-eks-7684af

Another thing that I noticed is that a lot of same jobs are being created and all with the status Pending. I don't know if this is normal behavior or not but there are more then 200 of them and counting.. Output kubectl get jobs:

...
cron-foo-qa-1616598720           0/1           17m        17m
cron-foo-qa-1616598780           0/1           16m        16m
cron-foo-qa-1616598840           0/1           15m        15m
cron-foo-qa-1616598900           0/1           14m        14m
cron-foo-qa-1616598960           0/1           13m        13m
cron-foo-qa-1616599020           0/1           12m        12m
cron-foo-qa-1616599080           0/1           11m        11m
cron-foo-qa-1616599200           0/1           9m2s       9m2s
cron-foo-qa-1616599260           0/1           8m4s       8m4s
cron-foo-qa-1616599320           0/1           7m7s       7m7s
cron-foo-qa-1616599380           0/1           6m11s      6m12s
cron-foo-qa-1616599440           0/1           5m1s       5m1s
cron-foo-qa-1616599500           0/1           4m4s       4m4s
cron-foo-qa-1616599560           0/1           3m6s       3m6s
cron-foo-qa-1616599620           0/1           2m10s      2m10s
cron-foo-qa-1616599680           0/1           74s        74s
cron-foo-qa-1616599740           0/1                      2s

I see some scheduling happening if I'm correct when I inspect the events list. Output of kubectl get events --sort-by='.metadata.creationTimestamp':

...
3s          Warning   FailedScheduling                  pod/cron-foobar-prod-1616590260-vwqsk           0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-acc-1616590260-j29vx            0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-prod-1616569560-g8mn2           0/6 nodes are available: 6 Insufficient pods.
3s          Normal    Scheduled                         pod/cron-foobar-acc-1616560380-6x88z            Successfully assigned middleware/cron-foobar-acc-1616560380-6x88z to ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal
3s          Warning   FailedScheduling                  pod/cron-foobar-prod-1616596560-hx895         0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-prod-1616598180-vwls2         0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-qa-1616536260-vh7bl           0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-acc-1616571840-68l54            0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-qa-1616564760-4wg7l           0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-prod-1616571840-7wmlc           0/6 nodes are available: 6 Insufficient pods.
3s          Normal    Started                           pod/cron-foobar-prod-1616564700-6gk58         Started container cron
3s          Warning   FailedScheduling                  pod/cron-foobar-acc-1616587260-hrcmq            0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-qa-1616595720-x5njq           0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-acc-1616525820-x5vhr            0/6 nodes are available: 6 Insufficient pods.
3s          Warning   FailedScheduling                  pod/cron-foobar-qa-1616558100-x4p96           0/6 nodes are available: 6 Insufficient pods.

Can someone help me in the right direction?

kubernetes-pod kubernetes scheduler
3 Answers

But the pod stays in the Pending state.

The pod has an event listed that it can't be scheduled because there are no nodes available.

This is as expected if you have reached your capacity. You can check the capacity of any node with:

kubectl describe node <node_name>

And to get a node name, use:

kubectl get nodes

To mitigate this, use more nodes, or fewer pods or configure so that the cluster can autoscale when this happens.

user_2132690

Things to try :

  • Remove all pods created by the job. Using kubectl delete --all pods --namespace=foo you can delete all the pods in the specified namespace. Also, maybe remove the job that evidently lacks configuration. The job can be configured to stop spawning pods after a defined number of failure or success. Check backOffLimit and restartPolicy in the kubernetes job documentation.

  • Check taints and tolerations. Describe your nodes with kubectl describe node <node_name> and the Taints: section inside. If there is some taints you will have to reflect this in the tolerations of your job. Also check for "memoryPressure" or things like that. It will also be listed in the node description.

  • Check available ressources with kubectl top nodes. Check available RAM and CPU.

  • Check if container image can be pulled. Maybe pull it on Docker and make sure it works and it's not giving you a timeout.

  • Check all networkPolicies with kubectl get netpol -A to ensure that any policies would block communication with pods in kube-system.

  • You can also check RBAC configuration but that would be far-fetched

user_55124550

What you can do first is check your cluster autoscaler, RBAC and also the role attached to your nodes(you might me missing some permissions there probabily.) For the crons check restartPolicy .

user_111970290
Related