How to add taint to AWS EKS node Group?

Viewed 5967

I am having one AWS EKS Cluster up & running. I need to add node group with taint. So that I can deploy the pod on particular node group in EKS. I can do it in azure AKS using the following command.

az aks nodepool add --resource-group rg-xx --cluster-name aks-xxx --name np1 --node-count 1 --node-vm-size xxx --node-taints key=value:NoSchedule --no-wait

How to achieve same in AWS EKS?

6 Answers

if you are using eksctl with managed nodegroups, you can patch the bootstrap script to achieve what you want.

managedNodeGroups:
  - name: batch
    preBootstrapCommands:
      - sed -i '/^KUBELET_EXTRA_ARGS=/a KUBELET_EXTRA_ARGS+=" --register-with-taints=tier=batch:NoSchedule"' /etc/eks/bootstrap.sh
    tags:
      k8s.io/cluster-autoscaler/node-template/taint/tier: "batch:NoSchedule"

this relies on the code at https://github.com/awslabs/amazon-eks-ami/blob/189baaa77c14120a1b62c42bacced17ba429466b/files/bootstrap.sh#L107 not changing too much.

You can similarly enable the docker network bridge (aka --enable-docker-bridge)

    preBootstrapCommands:
      - sed -i 's/^ENABLE_DOCKER_BRIDGE=.*/ENABLE_DOCKER_BRIDGE=true/' bootstrap.sh

The tracking issue for actual EKS support is at https://github.com/aws/containers-roadmap/issues/864, and this solution comes from a comment

I haven't seen eksctl as a requirment in the question so I'm adding another 2 options.

Option 1 - AWS CLI

From here:

Kubernetes node taints can be applied to new and existing managed node groups using the AWS Management Console or through the Amazon EKS API

The following is an example of creating a node group with a taint using the AWS CLI:

aws eks create-nodegroup \
 --cli-input-json '
{
  "clusterName": "my-cluster",
  ...
  "taints": [
     {
         "key": "dedicated",
         "value": "gpuGroup",
         "effect": "NO_SCHEDULE"
     }
   ],
}'

Option 2 - Terraform

If you're using Terraform then Hashicorp AWS provider supports taints out of the box with the taint configuration block:

resource "aws_eks_node_group" "stateless-ng" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "stateless-ng"
  .
  .
  .
  
  # Block 1 
  taint {
    key = "stateless-no-schedule"
    value  = "true"
    effect = "NO_SCHEDULE"
  }

  # Block 2    
  taint {
    key = "stateless-no-execute"
    value  = "true"
    effect = "NO_EXECUTE"
  }
}

Notice that each taint needs to be configured in a sperated configuration block.

You can Add like this in AWS:EKS:NodeGroup Cloudformation Template

Taints:
  - Effect: "NO_SCHEDULE"
    Key: !Ref NodeGroupName
    Value: "dedicated"

For those who are looking for a solution Using eksctl + yaml file

You can run the following command

eksctl create nodegroup --config-file=<clusterconfig.yaml>

<clusterconfig.yaml> is the file that you used to create your cluster. (Yes you can use the same file to create a new nodegroup/managednodegroup.) Add the new new managednodegroup and run the create command. eksctl will take a diff and create the new nodegroup. My cluster config looks like the yaml I have pasted below. I created nodegroup ng-system during the creation of the cluster. I later added a new cluster group ng-kafka, with taints, through the command I mentioned above.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: test-cluster
  region: us-east-2
  version: "1.22"
managedNodeGroups:
  - name: ng-system
    instanceType: t3.medium
    minSize: 3
    maxSize: 10
    spot: true
    maxPodsPerNode: 35
    labels: {
      role: system
    }
  - name: ng-kafka
    instanceType: m5.large
    desiredCapacity: 2
    minSize: 1
    maxSize: 4
    spot: true
    taints:
      - {
          "key": "dedicated",
          "value": "kafka",
          "effect": "NoSchedule"
        }
    ssh:
      allow: true
    labels: {
      role: kafka
    }
Related