How to set node taints using Terraform for Amazon EKS

Viewed 5719
4 Answers

In the Terraform script for that provider, you can add the following to a worker group:

(in main.tf)

worker_groups = [
  {
      name                          = "my_node_group"
      instance_type                 = "t3.medium"
      asg_desired_capacity          = 1
      asg_min_size                  = 1
      additional_security_group_ids = [aws_security_group.all_worker_mgmt.id]
      kubelet_extra_args            = "--node-labels=my_node_label --register-with-taints=my_node_label:NoSchedule"
      asg_max_size                  = 1
      tags = []
    },
]

The important part is to set kubelet_extra_args to apply a node label to the node and to use that node label to set the taint using --register-with-taints. These are commands run by EKS on each worker node at startup. (Note that all the other parameters I have set in the worker group can be changed based on your requirement)

You can check taints on nodes by using kubectl describe node <node_ip>.

Now eks module does support taints out of the box, just use the following configuration:

module "eks" {
  source = "terraform-aws-modules/eks/aws"
  ...
  node_groups = {
    test_name = {
      ...
      taints = [
        {
          key    = "dedicated"
          value  = "statefulset"
          effect = "NO_SCHEDULE"
        }
      ] 
    }
  }
}

Notice that you don't have to use a dedicated module for this feature because Hashicorp AWS provider supports taints out of the box with the taint configuration block:

resource "aws_eks_node_group" "statefulset-ng" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "statefulset-ng"
  .
  .
  .
  
  # Block 1 
  taint {
    key = "statefulset-no-schedule"
    value  = "true"
    effect = "NO_SCHEDULE"
  }

  # Block 2    
  taint {
    key = "statefulset-no-execute"
    value  = "true"
    effect = "NO_EXECUTE"
  }
}

Notice that each taint needs to be configured in a sperated configuration block.

This is how I created my node group with a taint using aws_eks_node_group resource with a pre-created EKS cluster.

resource "aws_eks_node_group" "test" {
  cluster_name    = var.cluster_name
  node_group_name = "test"
  node_role_arn   = master.worker_iam_role_arn
  subnet_ids      = var.vpc.private_subnets_id
  disk_size       = 20
  taint {
    key = "dedicated"
    value = "gpuGroup"
    effect = "NO_SCHEDULE"
  }
  scaling_config {
      desired_size = 1
      max_size     = 3
      min_size     = 1
  }
  labels = {
        "some-labels" = "labels"
  }
  instance_types = ["t3.micro"]
  remote_access {
    ec2_ssh_key = ssh-key.key_name
  }
}

Reference: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group#taint

Related