How to securely allow access to AWS Secrets Manager with Terraform and cloud-init

You can create aws_iam_policy or an inline policy which can allow access to certain SSM parameters based on date and time.

In case of inline policy, this can be attached to the instance role which would look something like this:

resource "aws_iam_role" "my_instance_iam_role" {
  name = "my_instance_iam_role"
  path = "/development/"

  inline_policy {
    name = "my_inline_policy"

    policy = jsonencode({
       "Version": "2012-10-17",
       "Statement": [{
           "Effect": "Allow",
           "Action": "ssm:GetParameters",
           "Resource": "arn:aws:ssm:us-east-2:123456789012:parameter/development-*",
           "Condition": {
               "DateGreaterThan": {"aws:CurrentTime": "2020-04-01T00:00:00Z"},
               "DateLessThan": {"aws:CurrentTime": "2020-06-30T23:59:59Z"}
           }
       }]
    })
  }
  tags = {
    Environment = "dev"
  }
}

So in the end the suggestions from @ervin-szilagyi got me 90% of the way there... I then needed to make some small changes to his suggestion. I am including my updated changes here to hopefully help others who struggle with this.

My aws_iam_role that allows temporary access (10 minutes) to the password now looks like:

resource "aws_iam_role" "my_instance_iam_role" {
  name = "my_instance_iam_role"
  path = "/development/"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      },
    ]
  })

  inline_policy {
    name = "access_my_password_iam_policy"

    policy = jsonencode({
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "secretsmanager:GetResourcePolicy",
            "secretsmanager:GetSecretValue",
            "secretsmanager:DescribeSecret",
            "secretsmanager:ListSecretVersionIds"
          ],
          "Resource": aws_secretsmanager_secret.my_password_secret.arn,
          "Condition": {
             "DateGreaterThan": { "aws:CurrentTime": timestamp() },
             "DateLessThan": { "aws:CurrentTime": timeadd(timestamp(), "10m") }
          }
        },
        {
            "Effect": "Allow",
            "Action": "secretsmanager:ListSecrets",
            "Resource": "*"
        }
      ]
    })
  }

  tags = {
    Environment = "dev"
  }
}

To retrieve the password during cloud-init, in the end I switched to using the aws CLI command as opposed to cURL, which yielded a cloud-init config like the following:

data "cloudinit_config" "my_instance_1" {
  gzip = true
  base64_encode = true

  part {
    content_type = "text/x-shellscript"
    filename = "setup-script.sh"
    content = <<EOF
#!/usr/bin/env bash
# Retrieve SA password from AWS Secrets Manager
command="aws --output text --region ${local.aws_region} secretsmanager get-secret-value --secret-id ${aws_secretsmanager_secret.my_password_secret.id} --query SecretString"
max_retry=5
counter=0
until my_password=$($command)
do
   sleep 1
   [[ counter -eq $max_retry ]] && echo "Failed!" && exit 1
   echo "Attempt #$counter - Unable to retrieve AWS Secret, trying again..."
   ((counter++))
done
server_password=$my_password /opt/srv/bin/install.sh
EOF
  }
}

There are two main ways to achieve this:

• pass the value as is during the instance creation with terraform
• post-bootstrap invocation of some script

Your approach of polling it in the cloud-init is a hybrid one, which is perfectly fine, but I'm not sure whether you actually need to go down that route.

Let's explore the first option, where you do everything in terraform. We have two sub-options there depending on where you create the secret and the instance within the same terraform execution run (within the same folder in which the code resides) or it's a two step process, where you create the secret first, and then the instance, as it has a minor difference between the two on how to pass the secret value as a var to the script.

• Case A: in case they are created together:

You can pass the password directly to the script.

resource "random_password" "my_password" {
  length = 16
  lower = true
  upper = true
  number = true
  special = true
  override_special = "@#$%"
}

resource "aws_secretsmanager_secret" "my_password_secret" {
  name = "/development/my_password"
}

resource "aws_secretsmanager_secret_version" "my_password_secret_version" {
  secret_id     = aws_secretsmanager_secret.my_password_secret.id
  secret_string = random_password.my_password.result
}

data "cloudinit_config" "my_instance_1" {
  gzip = true
  base64_encode = true

  part {
    content_type = "text/x-shellscript"
    filename = "setup-script.sh"
    content = <<EOF
#!/usr/bin/env bash
server_password=${random_password.my_password.result} /opt/srv/bin/install.sh
EOF
  }
}

• Case B: if they are created in separate folders

You could use a data resource to get the secret value from terraform (the role with which you are deploying your terraform code will need permissions GetSecret)

data "aws_secretsmanager_secret_version" "my_password" {
   secret_id = "/development/my_password"
}

data "cloudinit_config" "my_instance_1" {
  gzip = true
  base64_encode = true

  part {
    content_type = "text/x-shellscript"
    filename = "setup-script.sh"
    content = <<EOF
#!/usr/bin/env bash
server_password=${data.aws_secretsmanager_secret_version.my_password.secret_string} /opt/srv/bin/install.sh
EOF
  }
}

In both cases you wouldn't need to assign SSM permissions to the EC2 instance profile attached to the instance, you won't need to use curl or other means in the script, and the password would not be part of your bash script.

It will be stored in your terraform state, so you should make sure that the access to it is restricted. Even with the hybrid approach where you are going to get the secret from the secret manager during the instance bootstrap, the password would still be stored in your state as you are creating that secret with resource "random_password" as per the Terraform Sensitive data in state.

Now, let's look at option 2. It is very similar to your approach, but instead of doing it in the user-data, you can use Systems Manager Run Command to start your installation script as a post-bootstrap step. Then depending on how do you invoke the script, whether it is present locally on the instance, or you are using a document with a State Manager you can either pass the secret to it as a variable again, or get it from the Secrets Manager with aws-cli or curl, or whatever you prefer (which will require the necessary level of IAM permissions).