terraform minor aws user_data change "forces replacement" - What Is The Best Resolution?

Viewed 3781

I have made a minor change (comments only) to my AWS EC2 user_data bootstrap bash script. Terraform has detected this, and now wants to replace the EC2 with a new one. But this is a live service, and I'd rather not have to take it down, backup all the data, build a new server, and then restore the data - just to keep terraform happy!

Is there a simpler way of fixing this, by somehow bringing the code/statefile/aws in line so that I can use terraform to make other changes to the environment, but not have to worry about blowing away this server?

I've even tried editing the user data from the AWS console to try to bring it inline with what I think the changes should be, but this hasn't worked.

For now I'm using -target for specific changes, but this is probably not sustainable in the long run.

2 Answers

You can instruct the provider to ignore the changes to the user_data argument for that resource with the lifecycle meta-argument:

resource "aws_instance" "this" {
  # ...

  lifecycle {
    ignore_changes = [user_data]
  }
}

Per this issue https://github.com/hashicorp/terraform-provider-aws/issues/23315 the behavior now depends on https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance#user_data_replace_on_change which defaults to false so that it will not force replacement.

I had the issue however that I wanted the user_data script to execute on a Terraform apply, so I created a script that would do this on update of that field.

Note that this expects to be run in an environment with 1) an AWS_PROFILE set with permissions to execute a Session Manager command, 2) Session Manager Plugin installed, and 3) the target instance must have SSM Agent installed (see this link)

In the Terraform:

resource "null_resource" "update-user-data" {
  depends_on = [aws_instance.this]

  triggers = {
    user_data_base64 = base64encode(data.template_file.user_data.rendered)
  }
  provisioner "local-exec" {
    command     = file("${path.module}/run-remote-shell-script.sh")
    interpreter = ["bash", "-c"]
    environment = {
      INSTANCE_NAME = var.name
    }
  }
}

run-remote-shell-script.sh:

#!/usr/bin/env bash

test -n "$INSTANCE_NAME" || (echo missing INSTANCE_NAME; exit 1)
test -n "$AWS_PROFILE" || (echo missing AWS_PROFILE; exit 1)

INSTANCE_ID=$(aws ec2 describe-instances \
 --filters Name=tag:Name,Values=$INSTANCE_NAME \
  --query 'Reservations[].Instances[].InstanceId' --output text)

echo $INSTANCE_ID

aws ec2 wait instance-running --instance-ids $INSTANCE_ID;
aws ssm send-command --instance-ids $INSTANCE_ID \
   --document-name AWS-RunShellScript --parameters '{"commands": ["sudo bash /var/lib/cloud/instance/user-data.txt"]}'
sleep 1

exit 0
Related