Why are my pods not utilizing multiple nodes?

user_2656747 0
Viewed 365

Currently, I have 2 nodes on AWS

NAME                                           STATUS   ROLES    AGE   VERSION
NODE-1-xxx-xxx-xxx-xxx.cn-north-1.compute.internal   Ready    <none>   15d   v1.16.13-eks-2ba888
NODE-2-xxx-xxx-xxx-xxx.cn-north-1.compute.internal   Ready    <none>   13d   v1.16.13-eks-2ba888

Here is also a screenshot of my CPU loads

NODE 1 NODE !

NODE 2 NODE 2

My problem is whenever I deploy my application to production I will max out my cpu usage on NODE 2 and it will slow down the entire site

Here is my deployment config

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend # name of the deployment
  namespace: backend
  labels: # these labels apply to the deployment
    app: root
    component: backend

spec:
  replicas: 2
  minReadySeconds: 20
  selector:
    matchLabels:
      app: root
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels: # these labels apply to our container
        app: root
        component: backend
        version: xxx_${BITBUCKET_BUILD_NUMBER}
    spec:
      containers:
      - name: backend # name of our container
        image: xxx/xxx_main:${BITBUCKET_BUILD_NUMBER} # the URI that we got from ECR
        imagePullPolicy: Always
        envFrom:
        - configMapRef:
            name: env
        ports:
        - containerPort: 3000 # expose the running contianer on port 3000
          name: backend
          protocol: TCP
        readinessProbe:
          tcpSocket:
            port: backend
          initialDelaySeconds: 20
          periodSeconds: 5
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 20

      imagePullSecrets:
       - name: xxx

Am I not scaling things out properly here? what is the point of having two nodes if only one is ever in use? How can i properly scale my applications to use multiple nodes?

amazon-eks kubernetes amazon-web-services
1 Answers

This happens because the node scheduling algorithm is based on priority score with different priority algorithms contributing to the score. One such priority algorithm is the ImageLocalityPriority which adds a positive priority score for nodes already having the images used by the pod. So initially, a node that already has the first replica of the pod running, gets a small priority bump due to the ImageLocalityPriority. Once more and more replicas are added, the number of pods running on each node even out because other priorities like BalancedResourceAllocation etc also take affect.

There is also a SelectorSpreadPriority which helps in minimising the number of pods belonging to a same service on a node. So if you create your service object before creating the deployment, it might help.

To enforce the pods to spread out, you should add inter-pod anti-affinity constraints to your pods.

You should also consider adding requests and limits to your containers. This helps in spreading out the pods as the LeastRequestedPriority priority also kicks in. (Note: There is also MostRequestedPriority which adds priority for used nodes but it is not enabled by default).

user_54592010
Related