Can terraform run "apply" for multiple infrastructure/workspace in parallel?

user_13536432 0
Viewed 2794

We have one terraform instance and script which could create infra in azure. We would like to use same scripts to create/update/destroy isolated infra for each one of our customers on azure . We have achieved this by assigning one workspace for each client,different var files and using backend remote state files on azure.

Our intend is to create a wrapper python program that could create multiple threads and trigger terraform apply in parallel for all workspaces. This seems to be not working as terraform runs for one workspace at a time. Any suggestions/advice on how we can achieve parallel execution of terraform apply for different workspaces?

terraform0.12+ terraform azure python
1 Answers

It's safe to run multiple Terraform processes concurrently as long as:

  • They all have totally distinct backend configurations, both in terms of state storage and in terms of lock configuration. (If they have overlapping lock configuration then they'll mutex each other, effectively serializing the operations in spite of you running multiple copies.)
  • They work with an entirely disjoint set of remote objects, including those represented by both managed resources (resource blocks) and data resources (data blocks).

Most remote APIs do not support any sort of transaction or mutex concept directly themselves, so Terraform cannot generally offer fine-grained mutual exclusion for individual objects. However, multiple runs that work with entirely separate remote objects will not interact with one another.

Removing a workspace (using terraform workspace delete) concurrently with an operation against that workspace will cause undefined behavior, because it is likely to delete the very objects Terraform is using to track the operation.

There is no built-in Terraform command for running multiple operations concurrently, so to do so will require custom automation that wraps Terraform.

user_2818480
Related