Referencing gitlab secrets in Terraform

Question

I am quite new to Terraforms and gitlab CI and there is something that I am trying to do here with it.

I want to use Terraform to create an IAM user and a S3 bucket. Using policies allow certain operations on this S3 bucket to this IAM user. Have the IAM user's credentials saved in the artifactory. Now the above is going to be my core module.

The core module looks something like the below:

Contents of : aws-s3-iam-combo.git

(The credentials for the IAM user using which all the Terraform would be run, say admin-user, would be stored in gitlab secrets.)

main.tf

resource "aws_s3_bucket" "bucket" {
  bucket = "${var.name}"
  acl = "private"
  force_destroy = "true"

  tags {
    environment = "${var.tag_environment}"
    team        = "${var.tag_team}"
  }

  policy =<



outputs.tf

output "bucket" {
  value = "${aws_s3_bucket.bucket.bucket}"
}

output "bucket_id" {
  value = "${aws_s3_bucket.bucket.id}"
}

output "iam_access_key_id" {
  value = "${aws_iam_access_key.s3.id}"
}

output "iam_access_key_secret" {
  value = "${aws_iam_access_key.s3.secret}"
}


variables.tf

variable "name" {
  type = "string"
}

variable "tag_team" {
  type = "string"
  default = ""
}

variable "tag_environment" {
  type = "string"
  default = ""
}

variable "versioning" {
  type = "string"
  default = false
}

variable "profile" {
  type = "string"
  default = ""
}


Anyone in the organization who now needs to create S3 buckets, would need to create a new repo, something of the form:

main.tf

module "aws-s3-john-doe" {
  source = "git::https://git@gitlab-address/terraform/aws-s3-iam-combo.git?ref=v0.0.1"
  name = "john-doe"
  tag_team = "my_team"
  tag_environment = "staging"
}


gitlab-ci.yml

stages:
  - plan
  - apply

plan:
  image: hashicorp/terraform
  stage: plan
  script:
    - terraform init
    - terraform plan

apply:
  image: hashicorp/terraform
  stage: apply
  script:
    - terraform init
    - terraform apply
  when: manual
  only:
    - master


And then the pipeline would trigger and when this repo gets merged to master, the resources (S3 and IAM user) would be created and the user would have this IAM user's credentials. 

Now the problem is that we have multiple AWS accounts. So say if a dev wants to create an S3 in a certain account, it would not be possible with the above set up as the admin-user, whose creds are in gitlab secrets, is only for one account alone. 

Now I don't understand how do I achieve the above requirement of mine. I have the below idea: (Please suggest if there's a better way to do this)


Have multiple different creds set up in gitlab secrets for each AWS account in question
Take user input, specifying the AWS account they want the resources created in, as a variable. So something like say:


main.tf

module "aws-s3-john-doe" {
  source = "git::https://git@gitlab-address/terraform/aws-s3-iam-combo.git?ref=v0.0.1"
  name = "john-doe"
  tag_team = "my_team"
  tag_environment = "staging"
  aws_account = "account1"
}



And then in the aws-s3-iam-combo.git main.tf somehow read the creds for account1 from the gitlab secrets.


Now I do not know how achieve the above, like how do i read from gitlab the required secret variable etc. 

Can someone please help here?

user_10975889 · Answer

you asked this some time ago, but maybe my idea still helps the one or the other...

You can do this with envsubst (requires the pkg gettext to be installed on your runner or in the Docker image used to run the pipeline).

Here is an example:

First, in the project settings you set your different user accounts as environment variables (project secrets:

SECRET_1: my-secret-1
SECRET_2: my-secret-2
SECRET_3: my-secret-3

Then, create a file that holds a Terraform variable, let's name it vars_template.tf:

variable "gitlab_secrets" {
    description = "Variables from GitLab"
    type = "map"
    default = {
        secret_1 = "$SECRET_1"
        secret_2 = "$SECRET_2"
        secret_3 = "$SECRET_3"
    }
}

In your CI pipeline, you can now configure the following:

plan:dev:
  stage: plan dev
  script:
    - envsubst < vars_template.tf > ./vars_envsubst.tf
    - rm vars_template.tf
    - terraform init
    - terraform plan -out "planfile_dev"
  artifacts:
    paths:
      - environments/dev/planfile_dev
      - environments/dev/vars_envsubst.tf

apply:dev:
  stage: apply dev
  script:
    - cd environments/dev
    - rm vars_template.tf
    - terraform init
    - terraform apply -input=false "planfile_dev"
  dependencies:
    - plan:dev

It's important to note that the original vars_template.tf has to be deleted, otherwise Terraform will throw an error that the variable is defined multiple times. You could circumvent this by storing the template file in a directory which is outside the Terraform working directory though. But from the Terraform state you can see that the variable values where correctly substituted:

"outputs": {
    "gitlab_secrets": {
        "sensitive": false,
        "type": "map",
        "value": {
            "secret_1": "my-secret-1",
            "secret_2": "my-secret-2",
            "secret_3": "my-secret-3"
        }
    }
}

You can then access the values with "${vars.gitlab_secrets["secret_1"]}" in your Terraform resources etc.

UPDATE: Note that this method will store the secrets in the Terraform state file, which can be a potential security issue if the file is stored in an S3 bucket for collaborative work with Terraform. The bucket should at least be encrypted. In addition, it's recommended to limit the access to the files with ACLs so that, e.g., only a user terraform has access to it. And, of course, a user could reveil the secrets via Terraoform outputs...

Referencing gitlab secrets in Terraform

1 Answers