How to test Terraform files

We recently open sourced Terratest, our swiss army knife for testing infrastructure code.

Today, you're probably testing all your infrastructure code manually by deploying, validating, and undeploying. Terratest helps you automate this process:

1. Write tests in Go.
2. Use helpers in Terratest to execute your real IaC tools (e.g., Terraform, Packer, etc.) to deploy real infrastructure (e.g., servers) in a real environment (e.g., AWS).
3. Use helpers in Terratest to validate that the infrastructure works correctly in that environment by making HTTP requests, API calls, SSH connections, etc.
4. Use helpers in Terratest to undeploy everything at the end of the test.

Here's an example test for some Terraform code:

terraformOptions := &terraform.Options {
  // The path to where your Terraform code is located
  TerraformDir: "../examples/terraform-basic-example",
}

// This will run `terraform init` and `terraform apply` and fail the test if there are any errors
terraform.InitAndApply(t, terraformOptions)

// At the end of the test, run `terraform destroy` to clean up any resources that were created
defer terraform.Destroy(t, terraformOptions)

// Run `terraform output` to get the value of an output variable
instanceUrl := terraform.Output(t, terraformOptions, "instance_url")

// Verify that we get back a 200 OK with the expected text
// It can take a minute or so for the Instance to boot up, so retry a few times
expected := "Hello, World"
maxRetries := 15
timeBetweenRetries := 5 * time.Second
http_helper.HttpGetWithRetry(t, instanceUrl, 200, expected, maxRetries, timeBetweenRetries)

These are integration tests, and depending on what you're testing, can take 5 - 50 minutes. It's not fast (though using Docker and test stages, you can speed some things up), and you'll have to work to make the tests reliable, but it is well worth the time.

Check out the Terratest repo for docs and lots of examples of various types of infrastructure code and the corresponding tests for them.

From my research this is a tough issue, since Terraform is not meant to be a full featured programming language and you are declaring what resources you want with Terraform, not how to build them, trying to unit-test doesn't really give you the assurance you are building resources how you'd like without actually running an apply. This makes attempts to unit-test feel more like a linting to me.

However, you could parse your HCL files with something like pyhcl, or parse you're plan files, however from my experience this was a lot of work for little benefit (but I could be missing an easier method!).

Here are some alternatives if you wanted to test the results of your terraform applys:

kitchen-terraform is a tool for writing Test Kitchen specs for your infrastructure.

kitchen-verifier-awspec helps bring together awspec and kitchen-terraform, although I have not used it personally.

If you are using AWS, I have found AWS Config to be able to provide a lot of the same benefits as other infrastructure testing tools, without as much setup/maintenance. Although it is fairly new, and I have not used it extensively.

Also if you are paying for Terraform Premium you get access to Sentinel, which seems to provide a lot of similar benefits to AWS Config, however I have not used it personally.

In addition to the answers, I will add my two cents. I was not very happy using GO lang with Terratest although it works perfectly well. It is just that GO is not my favorite programming language. I looked for some frameworks in Java and I found terraform-maven. At first glance, I only found examples in Groovy, but since Groovy run on JVM, it is feasible to implement the same examples in Java.

I translated part of the S3PreProvisionSpec.groovy to Java. It is testing this main.tf file.

@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class S3PreProvisionTest {

    private final String TF_CHANGE = "change";
    private final String TF_AFTER = "after";
    private final String TF_TAGS = "tags";
    private final Map<String, String> mandatoryTags = Map.of(
        "application_id", "cna",
        "stack_name", "stacked",
        "created_by", "f.gutierrez@yieldlab.de"
    );
    private Terraform terraform;
    private TfPlan tfplan;

    @BeforeAll
    void setup() {
        terraform = new Terraform().withRootDir("s3_pre_post_demo")
        // .withProperties(Map.of("noColor", "true"))
        ;
        tfplan = terraform.initAndPlan();
    }

    @AfterAll
    void cleanup() {
        terraform.destroy();
    }

    @Test
    void mandatoryTagsForS3Resources() {
        List<Map> s3Bucket = tfplan.getResourcesByType("aws_s3_bucket");
        System.out.println("=========================");

        s3Bucket.forEach(map -> {
            Map tfChangeMap = (Map) map.get(TF_CHANGE);
            Map tfAfterMap = (Map) tfChangeMap.get(TF_AFTER);
            Map tfTagsMap = (Map) tfAfterMap.get(TF_TAGS);

            assertEquals(3, tfTagsMap.size());
            mandatoryTags.forEach((k, v) -> {
                assertEquals(v, tfTagsMap.get(k));
            });

            try {
                JSONObject jsonObject = new JSONObject(map);
                JSONObject jsonChange = jsonObject.getJSONObject(TF_CHANGE);
                JSONObject jsonAfter = jsonChange.getJSONObject(TF_AFTER);
                JSONObject jsonTags = jsonAfter.getJSONObject(TF_TAGS);
                System.out.println(">>>>>>>>>>>>>>>>>>>> " + jsonTags.toString());
                mandatoryTags.forEach((k, v) -> {
                    try {
                        assertEquals(v, jsonTags.getString(k));
                    } catch (JSONException e) {
                        e.printStackTrace();
                    }
                });
            } catch (JSONException e) {
                e.printStackTrace();
            }
        });
    }
}

One approach is to output the results to a file using -out=tempfile, then run a script to validate whatever you're trying to do, and if all passes you can pass the file into the apply command. look at -out here: https://www.terraform.io/docs/commands/plan.html

You can use github.com/palantir/tfjson to parse a .plan file to json.

There is an issue at the moment that give a "unknown plan file version: 2" error. This is because the vendored version of terraform is too old.

The fix is:

go get github.com/palantir/tfjson
cd $GOPATH/src/github.com/palantir/tfjson
rm -rf vendor
go get -v ./...

There is then an error in ../../hashicorp/terraform/config/testing.go. To fix just change the line

t.Helper()

to

//t.Helper()

Run go get again and then go install

go get -v ./...
go install ./...

You should then be able to do the following which will produce json output.

terraform plan --out=terraform.plan 
tfjson terraform.plan