How do I implement a retry pattern with Terraform?

Question

My use case: I need to create an AKS cluster with Terraform azurerm provider, and then set up a Network Watcher flow log for its NSG.

Note that as many other AKS resources, the corresponding NSG is not controlled by Terraform. Instead, it's created by Azure indirectly (and asynchronously), so I treat it as data, not resource.

Also note that Azure will create and use its own NSG even if the AKS is created with a customary created VNet.

Depending on the particular region and the Azure API gateway, my team has seen up to 40 minute delay between having the AKS created and then the NSG resource visible in the node pool resource group.

If I don't want my Terraform config to fail, I see 3 options:

Run a CLI script that waits for the NSG, make it a null_resource and depend on it
Implement the same with a custom provider
Have a really ugly workaround that implements a retry pattern - below is 10 attempts at 30 seconds each:

data "azurerm_resources" "my_nsg_1" {
  resource_group_name = var.clusterNodeResourceGroup
  type = "Microsoft.Network/networkSecurityGroups"
}
resource "time_sleep" "my_nsg_sleep1" {
  count = length(data.azurerm_resources.my_nsg_1.resources) == 0 ? 1 : 0
  create_duration = "30s"
  triggers = {
    ts = timestamp()
  }  
}
data "azurerm_resources" "my_nsg_2" {
  depends_on = [time_sleep.my_nsg_sleep1]
  resource_group_name = var.clusterNodeResourceGroup
  type = "Microsoft.Network/networkSecurityGroups"
}
resource "time_sleep" "my_nsg_sleep2" {
  count = length(data.azurerm_resources.my_nsg_1.resources) == 0 ? 1 : 0
  create_duration = length(data.azurerm_resources.my_nsg_2.resources) == 0 ? "30s" : "0s"
  triggers = {
    ts = timestamp()
  }  
}
...
data "azurerm_resources" "my_nsg_11" {
  depends_on = [time_sleep.my_nsg_sleep10]
  resource_group_name = var.clusterNodeResourceGroup
  type = "Microsoft.Network/networkSecurityGroups"
}

// Now azurerm_resources.my_nsg_11 is OK as long as the NSG was created and became visible to the current API Gateway within 5 minutes.

Note that Terraform doesn't allow resource repeating via the use of "for_each" or "count" at more than an individual resource level. In addition, because it resolves dependencies during the static phase, two sets of resource lists created with "count" or "for_each" cannot have dependencies at an individual element level of each other - you can only have one list depend on the other, obviously with no circular dependencies allowed.

E.g. my_nsg[count.index] cannot depend on my_nsg_delay[count.index-1] while my_nsg_delay[count.index] depends on my_nsg[count.index]

Hence this horrible non-DRY antipattern.

Is there a better declarative solution so I don't involve a custom provider or a script?

How do I implement a retry pattern with Terraform?

0 Answers