Kubernetes doesn't schedule pod to the node with less workload

Question

I use AKS to create our cluster with 3 worker node and their names are node-0, node-1, node-2. They are created by Azure VMSS.

When I deploy a pod-new through Helm, the scheduler always schedules pod-new to node-0 or node-2 even though those node with heavy workload.

Assume there is a pod-old on node-0 or node-2, It will result in pod-old be evicted but still doesn't re-scheduled to node-1. I need to explicit use nodeSelector to move it to node-1.

Below is my deployment.yaml. To gain better resource usage, I don't specify request and limit cause my pod needs higher cpu-burst to do initialize. After initialization, it only needs a little resource to serve request.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-service
  labels:
    helm.sh/chart: service-0.1.0
    app.kubernetes.io/name: service
    app.kubernetes.io/instance: aks
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  minReadySeconds: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: service
      app.kubernetes.io/instance: aks
  template:
    metadata:
      labels:
        app.kubernetes.io/name: service
        app.kubernetes.io/instance: aks
    spec:
      imagePullSecrets:
        - name: custom
      serviceAccountName: account-with-azure-cr
      securityContext:
        fsGroup: 2000
        runAsGroup: 3000
        runAsUser: 1000
      containers:
        - name: frontend
          image: imageurl
          imagePullPolicy: IfNotPresent
          ports:
            - name: port-frontend
              containerPort: 3000
              protocol: TCP
      terminationGracePeriodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlogpod
          hostPath:
            path: /var/log/pods

user_174145 · Answer

In order for kubernetes to understand the resource utilization, requests data is used. You can read about best practices mentioned by AKS from link.

Requests tells the kubernetes to block some resources for running the pods. I guess when you are saying provisioning of service, you are talking about pods only as pod is the working unit here (service is the load balancer abstraction which redirect the traffic to appropirate pod). Even if your pod requires very less resources, you can mention them in requests. Limit is something which k8s will enforce (so that is not required in your case) but requests should be must.

Quoting from above document

In your pod specifications, it's best practice and very important to define these requests and limits based on the above information. If you don't include these values, the Kubernetes scheduler cannot take into account the resources your applications require to aid in scheduling decisions.

Kubernetes doesn't schedule pod to the node with less workload

1 Answers