How to integrate terraform in GitLab CI/CD

LP13 0
Viewed 32

I have terraform scripts that creates resources. Each script is in different folder for the separation of concern. When I manually apply these script, I navigate to each folder and then use terraform apply.

1>cd c:\tf\kms\terraform apply
2>cd c:\tf\roles\terraform apply
3>cd c:\tf\lambda\terraform apply

for each apply I review the plan and then type yes to apply changes.

Now I have to make this part of GitLab CI/CD. But what I am missing here, in pipeline how do I apply changes? because apply will need manual interaction to type yes

Terraform has -auto-approve flag but I think that is very dangerous.

UPDATE1
Since the resources are organized in multiple folders in the same repository and they all need to be created using same pipeline, How do I arrange the plan out

Will the following pipeline create 3 plans inside planoutput folder in artifacts?

stages:
 - plan
 
 
image: "docker/mycompany/terraform"

plan:
  stage: plan  
  script:
    - chmod +x ./scripts/dev.sh
    - ./scripts/dev.sh
artifacts:    
  paths:
    - planoutput

and then in scripts/dev.sh

#!/bin/bash

cd ${CI_PROJECT_DIR}/tf/kms/
terraform init
terraform validate
terraform plan -out=kms.tfplan

cd ${CI_PROJECT_DIR}/tf/roles/
terraform init
terraform validate
terraform plan -out=roles.tfplan

cd ${CI_PROJECT_DIR}/tf/lambda/
terraform init
terraform validate
terraform plan -out=lambda.tfplan
terraform-provider-aws gitlab-ci-runner gitlab gitlab-ci terraform
1 Answers

I would suggest a three-stage pipeline:

  • Stage 1. Validate the terraform config (terraform validate, tflint, regula or conftest)
  • Stage 2. Generate a plan
  • Stage 3. Apply the changes

Stages 1 and 2 should be automatic when a checkin is pushed, stage three would need an authorised human to press a button after reviewing the plan.

There are edge cases that you would need to be careful about, and you need to design the pipelines so that it's not possible to run in parallel. I would recommend that you also use a pull request process so that only after someone has approved the changes and merged the PR can the new Terraform be applied but that is up to you really.

It's quite important that the apply (stage 3) applies the plan as reported in stage 2 rather than regenerating the plan at that point otherwise it would be possible for the apply stage to do something different from the plan stage (if the underlying infrastructure changes for some reason). For this reason, you might also want to make the plan time out if it's not applied within the next hour, the pipeline forces you to rerun the plan.

There is also the question of state, where it is stored and who has access to it.

Update 1

With regard to the question of multiple folders each containing different resources, you state that this is done for the reason of separation of concern, if this is truly a different concern (as in belongs to a different project or changes due to different reasons or by different people) then they should be in a different pipeline, having separate folders that live in the same repo and are all either applied together or not is not getting any separation.

As i see it, you have two options:

  1. Create three repos so that you have true separation, each would have its own pipelines and you can assign different users with different permissions to each. you don't need to merge and apply all or nothing anymore.
  2. Put a main.tf in the root and call terraform once which will initialise all three at the same time

Personally, I have gone with option 1 in my projects, we have tens of pipelines each can be merged/applied independently and many people can work across the many projects without too much onerous coordination.

If you insist on keeping your current folder structure think about how you will store the state for the different folders, how you will enforce locking and the like. Also, what will you do when folder 1 is successfully applied but there is an error applying folder two? how will you recover from this? Personally, there are too many downsides with the setup you are attempting, simplify and make your life easier.

userRlmt0
Related