How to manage hundreds of AWS S3 buckets with Terraform

Question

Im looking to manage hundreds of AWS S3 buckets using Terraform and would like to create all of them by defining a variable list array of bucket names along with a few other s3 resource specific parameters, loop thru that list with a for_each and then use the results from that array to define the bucket configuration. The issue that we're running into are the lifecycle rules. Our buckets have several different lifecycle rules based om the bucket requirements, but out of say 1000 buckets, we have maybe 10 different lifecycle rule. Using the aws cli, its easy enough to manage bucket lifecycles, it is its own specific command:

aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle_1.json

and we can choose from a short list of lifecycle_X.json files to modify the rules.

The problem we're running into is that the lifecycle attribute in the terraform resource "aws_s3_bucket" is not, or cannot be a template? Apparently it was a template back in version 11, but since version 12 it has to be some block of code. What I'd like to do is something like this:

locals {
  lifecycle_rule_1 = [
    id      = "rule1"
    prefix  = ""
    enabled = true
    expiration = [ 
      {   
        days = 30
    }
  ]
  lifecycle_rule_2 = [
    id      = "rule2"
    prefix  = ""
    enabled = true
    expiration = [ 
      {   
        days = 60
    }
  ]
}

resource "aws_s3_bucket" "bucket" {
  for_each = locals.bucket_definitions
  bucket = "my-bucket"
  acl    = "private"
  lifecycle_rule = [locals.lifecycle_rule_1]
}

Just pinging here is anyone has come up with a similar problem and a solution? Thanks

user_248823 · Answer

Sadly, such functionality is not supported. There is already a future request for that on GitHub:

Feature Request: Pass complex blocks as variables

Thus, you have to define the components of lifecycle_rule "individually" as shown in the docs.

The alternative is to use a custom solution for setting up the lifecycles, for example using AWS CLI through local-exec or through AWS SDK using aws_lambda_invocation.

user_4522147 · Answer

You can create a map in your locals like so:

locals {
  lifecycle_rules = {
    lifecycle_rule_1 = {
      id      = "rule1"
      prefix  = ""
      enabled = true
      expiration = [ 
        {   
          days = 30
        }
      ]
    }
    lifecycle_rule_2 = {
      id      = "rule2"
      prefix  = ""
      enabled = true
      expiration = [
        {   
          days = 60
        }
      ]
    }
  }
}

and then use them like so:

resource "aws_s3_bucket" "bucket" {
  for_each = local.bucket_definitions
  bucket = each.value.bucket
  acl = each.value.acl
  lifecycle_rule {
    id = local.lifecycle_rules[each.value.lifecycle_rule_name].id
    prefix = local.lifecycle_rules[each.value.lifecycle_rule_name].prefix
    enabled = local.lifecycle_rules[each.value.lifecycle_rule_name].enabled
    expiration = local.lifecycle_rules[each.value.lifecycle_rule_name].expiration
  }
}

Essentially, you'd need to create the mapping between the exact same name in your local.bucket_definitions.lifecycle_rule_name to match with the key of the map defined above in local.lifecycle_rules.
Also, you might need to have some default values in your lifecycle_rules map, in case some of your rules don't use all the same attributes

How to manage hundreds of AWS S3 buckets with Terraform

2 Answers