I am trying to deploy an Autopilot cluster on GKE however I am getting insufficient CPU/memory errors as below when trying to deploy a Pod. Kubectl get nodes returns 3 nodes, each with about 0.5GB free cpu and same for memory so quite small. I am trying to run a GPU heavy job so I expect GKE to scale up however it fails saying insufficient resources. What am I doing wrong?
Warning FailedScheduling 27m (x5 over 31m) gke.io/optimize-utilization-scheduler 0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory.
Warning FailedScheduling 26m gke.io/optimize-utilization-scheduler 0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 2 Insufficient cpu, 2 Insufficient memory.
Normal TriggeredScaleUp 26m cluster-autoscaler pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/picdmo-342711/zones/us-central1-c/instanceGroups/gk3-picdmo-nap-1wcisjk4-2ba03e97-grp 0->1 (max: 1000)}]
Normal NotTriggerScaleUp 25m (x6 over 30m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 Insufficient cpu, 1 Insufficient memory, 2 in backoff after failed scale-up
Normal NotTriggerScaleUp 20m (x14 over 21m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 2 Insufficient cpu, 1 Insufficient memory
Normal TriggeredScaleUp 15m cluster-autoscaler pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/picdmo-342711/zones/us-central1-c/instanceGroups/gk3-picdmo-nap-xt7d8ijc-37c84d94-grp 0->1 (max: 1000)}]
Warning FailedScaleUp 15m (x5 over 31m) cluster-autoscaler Node scale up in zones us-central1-c associated with this pod failed: GCE quota exceeded. Pod is at risk of not being scheduled.
Warning FailedScheduling 15m (x6 over 20m) gke.io/optimize-utilization-scheduler 0/4 nodes are available: 3 Insufficient memory, 4 Insufficient cpu.
Normal NotTriggerScaleUp 14m (x2 over 15m) cluster-autoscaler (combined from similar events): pod didn't trigger scale-up (it wouldn't fit if a new node is added): 2 in backoff after failed scale-up, 2 Insufficient cpu, 1 Insufficient memory
Warning FailedScheduling 13m (x2 over 14m) gke.io/optimize-utilization-scheduler 0/4 nodes are available: 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1660665555}, that the pod didn't tolerate, 3 Insufficient cpu, 3 Insufficient memory.
Normal NotTriggerScaleUp 4m50s (x135 over 29m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 Insufficient cpu, 1 Insufficient memory
Warning FailedScheduling 92s (x17 over 25m) gke.io/optimize-utilization-scheduler 0/3 nodes are available: 3 Insufficient cpu, 3 Insufficient memory.```
apiVersion: batch/v1
kind: Job
metadata:
generateName: asd-job-
spec:
template:
spec:
containers:
- name: asd
image: gcr.io/-342711/-job:latest
imagePullPolicy: Always
command: ["/bin/sh"]
args: ["-c", "echo"]
resources:
requests:
memory: "16000Mi"
cpu: "8000m"
limits:
memory: "32000Mi"
cpu: "16000m"
nvidia.com/gpu: 2
restartPolicy: Never
backoffLimit: 4