Create multiple folders within multiple S3 buckets with Terraform

Question

I am looking to create one S3 terraform module which can take list of bucket names and folder names to be created inside all those buckets. For e.g. in my S3 module main.tf. I have

resource "aws_s3_bucket_object" "folders" {
    count = var.create_folders ? length(var.s3_folder_names) : 0
    bucket  = element(aws_s3_bucket.s3bucket.*.id, count.index)
    acl    = "private"
    key    = format("%s/",var.s3_folder_names[count.index])
    source = "/dev/null"
}

I am calling this module as given below:

variable "s3_bucket_name" {
  type = list
  description = "List of S3 bucket names"
default = ["bucket1","bucket-2"]
}
variable "s3_folder_names" {
  type        = list
  description = "The list of S3 folders to be created inside S3 bucket"
  default=["folder1/dev","folder2/qa"]

}
module "s3" {
  source = "../../../gce-nextgen-cloud-terraform-modules/modules/s3"
  create_folders = true
  s3_folder_names = var.s3_folder_names
  environment = var.environment
  s3_bucket_name = var.s3_bucket_name
  force_destroy = true
  bucket_replication_enabled = true
  tags = local.common_tags
  providers = {
    aws.main_region = aws.main_region
    aws.secondary_region = aws.secondary_region
  }
}

I am facing problem because count variable can only be set once in resource block. Here is the scenario that is cauing problems:

If var.s3_folder_names < aws_s3_bucket.s3bucket.*.id. Then I will not be able to access all the elements of S3 bucket list as shown below

resource "aws_s3_bucket_object" "folders" {
    count = var.create_folders ? length(var.s3_folder_names) : 0
    **bucket  = element(aws_s3_bucket.s3bucket.*.id, count.index)**
    acl    = "private"
    key    = format("%s/",var.s3_folder_names[count.index])
    source = "/dev/null"
}

Hence because of this I will not be able to create these folders inside all of the buckets. The only goal is to create same set of folder structure within all of those buckets. Any help would be truly appreciated. Thanks in advance!

user_248823 · Answer

You can create a combined data structure, e.g.:

locals {

  buckets_and_folders = merge([
    for bucket in var.s3_bucket_name:
      {
        for folder in var.s3_folder_names:
        "${bucket}-${folder}" => {
          bucket = bucket
          folder = folder
        }
      }
  ]...)

}

Then you would iterate over this structure using for_each:

resource "aws_s3_bucket_object" "folders" {

    for_each = var.create_folders ? local.buckets_and_folders : {}

    bucket  = each.value.bucket
    acl    = "private"
    key    = format("%s/", each.value.folder)
    source = "/dev/null"
}

Create multiple folders within multiple S3 buckets with Terraform

1 Answers