How to detect changes made outside of Terraform?

user_1781080 0
Viewed 1292

I have been using Terraform now for some months, and I have reached the point where my infrastructure is all base in Terraform files and I now have better control of the resources in our multiple accounts.

But I have a big problem. If someone makes a "manual" alteration of any Terraformed resource, it is easy to detect the change.

But what happens if the resource was not created using Terraform? I just don't know how to track any new resource or changes in them if the resource was not created using Terraform.

terraform
2 Answers

A key design tradeoff for Terraform is that it will only attempt to manage objects that it created or that you explicitly imported into it, because Terraform is often used in mixed environments where either some objects are managed by other software (like an application deployment tool) or the Terraform descriptions are decomposed into multiple separate configurations designed to work together.

For this reason, Terraform itself cannot help with the problem of objects created outside of Terraform. You will need to solve this using other techniques, such as access policies that prevent creating objects directly, or separate software (possibly created in-house) that periodically scans your cloud vendor accounts for objects that are not present in the expected Terraform state snapshot(s).

Access policies are typically the more straightforward path to implement, because preventing objects from being created in the first place is easier than recognizing objects that already exist, particularly if you are working with cloud services that create downstream objects as a side-effect of their work, as we see with (for example) autoscaling controllers.

user_2818480

Martin's answer is excellent and explains that Terraform can't be the arbiter of this as it is designed to play nicely both with other tooling and with itself (ie across different state files).

He also mentioned that access policies (although these have to be cloud/provider specific) are a good alternative to this so this answer will instead provide some options here for handling this with AWS if you do want to enforce this.

The AWS SDKs and other clients, including Terraform, all provide a user agent header in all requests. This is recorded by CloudTrail and thus you can search through CloudTrail logs with your favourite log searching tools to look for API actions that should be done via Terraform but don't use Terraform's user agent.

The other option that uses the user agent request header is to use IAM's aws:UserAgent global condition key which will block any requests that don't match the user agent header that's defined. An example IAM policy may look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1598919227338",
      "Action": [
        "dlm:GetLifecyclePolicies",
        "dlm:GetLifecyclePolicy",
        "dlm:ListTagsForResource"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Sid": "Stmt1598919387700",
      "Action": [
        "dlm:CreateLifecyclePolicy",
        "dlm:DeleteLifecyclePolicy",
        "dlm:TagResource",
        "dlm:UntagResource",
        "dlm:UpdateLifecyclePolicy"
      ],
      "Effect": "Allow",
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:UserAgent": "*terraform*"
        }
      }
    }
  ]
}

The above policy allows the user, group or role it is attached to to be able to perform read only tasks to any DLM resource in the AWS account. It then allows any client with a user agent header including the string terraform to perform actions that can create, update or delete DLM resources. If a client doesn't have terraform in the user agent header then any requests to modify a DLM resource will be denied.

Caution: It's worth noting that clients can override the user agent string and so this shouldn't be relied on as a foolproof way of preventing access to things outside of this. The above mentioned techniques are mostly useful to get an idea about the usage of other tools (eg the AWS Console) in your account where you would prefer changes to be made by Terraform only.

The AWS documentation to the IAM global condition keys has this to say:

Warning

This key should be used carefully. Since the aws:UserAgent value is provided by the caller in an HTTP header, unauthorized parties can use modified or custom browsers to provide any aws:UserAgent value that they choose. As a result, aws:UserAgent should not be used to prevent unauthorized parties from making direct AWS requests. You can use it to allow only specific client applications, and only after testing your policy.

The Python SDK, boto, covers how the user agent string can be modified in the configuration documentation.

user_22913210
Related