How can I name eks worker nodes provisioned with terraform?

user_7368808 0
Viewed 6345

I am using terraform 12.20.0 and I have provisioned an EKS cluster with 2 node groups.

How can I add name tags to EKS node workers according to their node group names?

I have tried adding "Name" tag in the additional tag sections of each node-group but the tags did not take and my EC2 instance names are empty, while other tags appear.

Here is the configuration - I have skipped the less relevant bits:

module "eks-cluster" {
...
  node_groups_defaults = {
    disk_size = 128
    key_name  = var.key_name
    subnets = [
      aws_subnet.{{}}.id,
      aws_subnet.{{}}.id,
    ]
    k8s_labels = {
      env = var.environment
    }

    additional_tags = {
      env                                             = var.environment
      "k8s.io/cluster-autoscaler/enabled"             = "true"
      "k8s.io/cluster-autoscaler/${var.cluster-name}" = "true"
    }
  }

  node_groups = {
    app = {
      name = "app"

      .....

      k8s_labels = {
        nodegroup = "app"
      }
      additional_tags = {
        nodegroup = "app"
        Name      = "${var.environment}-app-node"
      }
    }
    ml = {
      name = "ml"

     ...
      instance_type = "m5.xlarge"
      k8s_labels = {
        nodegroup = "ml"
      }
      additional_tags = {
        nodegroup = "ml"
        Name      = "${var.environment}-ml-node"
      }
    }
  }

  tags = {
    env = var.environment
  }

  map_roles = [{
    ......
  }]
}
amazon-eks terraform tags
5 Answers

As per documentation Resource: aws_eks_node_group doesn't allow for modifying tags on your instances.

There is a nice feature coming soon to EKS node groups which will allow you to pass a custom userdata script. Using that you will be able to modify programatically tags for your instances. Issues can be tracked -> https://github.com/aws/containers-roadmap/issues/596

UPDATE: As of 20/08/2020, you can now utilise launch_template with your node group. This will allow you to pass in Name tag. Example:

resource "aws_launch_template" "cluster" {
  image_id               = data.aws_ssm_parameter.cluster.value
  instance_type          = "t3.medium"
  name                   = "eks-launch-template-test"
  update_default_version = true

  tag_specifications {
    resource_type = "instance"

    tags = {
      Name                        = "eks-node-group-instance-name"
    }
  }
}
user_22758150

I just noticed that

"k8s.io/cluster-autoscaler/${var.cluster-name}" = "true"

Might need to be

"k8s.io/cluster-autoscaler/${var.cluster-name}" = "owned"
user_78117910

There is an existing issue with node group to add the "Name" tag on ASG. https://github.com/aws/containers-roadmap/issues/608 (open) and this on terraform end https://github.com/terraform-aws-modules/terraform-aws-eks/issues/860 (closed) However, there is an alternative to use aws cli command to add tag explicitly.

Try below to use in terraform

resource "null_resource" "add_custom_tags_to_asg" {  
  for_each = module.eks-cluster.node_groups

  provisioner "local-exec" {
    command = <<EOF
        aws autoscaling create-or-update-tags \
          --tags ResourceId=${each.value.resources[0].autoscaling_groups[0].name},ResourceType=auto-scaling-group,Key=Name,Value=k8s-node-groups-${each.value.labels["env"]},PropagateAtLaunch=true
    EOF
  }
}
user_30578430

The following Terraform resource works.

resource "aws_autoscaling_group_tag" "your_group_tag" {    
  autoscaling_group_name = aws_eks_node_group.your_group.resources[0].autoscaling_groups[0].name

  tag {
    key   = "Name"
    value = "enter-your-name-here"

    propagate_at_launch = true
  }

  depends_on = [
    aws_eks_node_group.your_group
  ]
}
user_119923030

Had the same issue, can this is the solution I came up with which works great.

First, create the ASG tags via the aws_autoscaling_group_tag resource.

resource "aws_autoscaling_group_tag" "mytag" {
  autoscaling_group_name = aws_eks_node_group.main.resources[0].autoscaling_groups[0].name

  tag {
    key                 = "foo"
    value               = "bar"
    propagate_at_launch = true
  }

  depends_on = [aws_eks_node_group.main]
}

Unfortunately this resource block doesn't accept multiple tags, so you'd have to create this resource block individually for each tag.

Another thing to keep in mind, is that the tags are applied to future scaled EC2 instances, not the currently running ones.

Which means, that you need to either manually scale down your nodes and scale back up, or write a bash script and run it as a local provisioner with terraform.

resource "null_resource" "refresh_autoscale" {
  provisioner "local-exec" {
    command = "cd ${path.module}/scripts ; bash ./scale_refresh.sh"
    environment = {
      ASG_NAME        = aws_eks_node_group.main.resources[0].autoscaling_groups[0].name
      CLUSTER_NAME    = "foo_cluster"
      NODE_GROUP_NAME = "foo_cluster_node"
      REGION          = var.region
      AWS_PROFILE     = var.aws_profile
      DESIRED_SIZE    = var.desired_size
      MIN_SIZE        = var.min_size
      MAX_SIZE        = var.max_size
    }
  }

  depends_on = [aws_eks_node_group.main]
}

Your bash script can run commands via the AWS CLI to scale down and up your node groups.

   aws --profile ${AWS_PROFILE} --region ${REGION} eks update-nodegroup-config --cluster-name ${CLUSTER_NAME} \
 --scaling-config "minSize=0,maxSize=1,desiredSize=0" --nodegroup-name ${NODE_GROUP_NAME}

Because the instances are not scaled down immediately, there is a period of waiting for the scale down to complete. If you have jq installed, you can periodically query the state of your ASG and see how many instances are currently running.

INSTANCE_COUNT=$(aws --profile ${AWS_PROFILE} --region ${REGION} autoscaling describe-auto-scaling-groups --auto-scaling-group-name ${ASG_NAME} \
| jq '.[][0] | .Instances | length')
user_63086850
Related