Application Gateway (or Load Balancer) with App Services using Terraform?

user_16920106 0
Viewed 1116

I am looking for the best way to implement load balancing (including redirection between multiple app services based on URL path) with App Services. This is what I have right now:

# App Service Plan
resource "azurerm_app_service_plan" "frontend" {
  name                = "${local.prefix}-frontend-asp"
  location            = azurerm_resource_group.frontend.location
  resource_group_name = azurerm_resource_group.frontend.name
  kind                = "Linux"
  reserved            = true
  app_service_environment_id = azurerm_app_service_environment_v3.frontend.id

  sku {
    tier = "Standard"
    size = "S1"
  }
}

# Environment
resource "azurerm_app_service_environment_v3" "frontend" {
  name                = "${local.prefix}-frontend-env"
  resource_group_name = azurerm_resource_group.frontend.name
  subnet_id           = azurerm_subnet.frontend.id

  cluster_setting {
    name  = "DisableTls1.0"
    value = "1"
  }

  cluster_setting {
    name  = "InternalEncryption"
    value = "true"
  }

  cluster_setting {
    name  = "FrontEndSSLCipherSuiteOrder"
    value = "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  }

  tags = {
    env         = "production"
    terraformed = "true"
  }
}

# Main App Service
resource "azurerm_app_service" "frontend" {
  name                = "${local.prefix}-frontend-app"
  location            = azurerm_resource_group.frontend.location
  resource_group_name = azurerm_resource_group.frontend.name
  app_service_plan_id = azurerm_app_service_plan.frontend.id

  site_config {
    linux_fx_version = "NODE|14"
    app_command_line = "pm2 serve /home/site/wwwroot --no-daemon --spa"
  }

  app_settings = local.front_env_variables
}

[I am using app_service_environment_id = azurerm_app_service_environment_v3.frontend.id as the best way of having App Service inside a Subnet controlled by me, to be able to control basic security via Network Security Group. I found out that azurerm_app_service_virtual_network_swift_connection is not really what I need since it only allows App Service to talk to other resources inside a particular subnet, not necessairly does it place an App Service in the subnet per se. Correct me if I'm wrong.]

# Resource Group
resource "azurerm_resource_group" "networking" {
  name     = "${local.prefix}-networking-rg"
  location = local.location
  provider = azurerm.uat
}

# Virtual Network
resource "azurerm_virtual_network" "vnet" {
  name                = "${local.prefix}-vnet"
  location            = azurerm_resource_group.networking.location
  resource_group_name = azurerm_resource_group.networking.name
  address_space       = ["10.1.0.0/16"]

  tags = {
    environment = local.prefix
  }
}

# Subnets for App Service instances
resource "azurerm_subnet" "frontend" {
  name                 = "frontend-app"
  resource_group_name  = azurerm_resource_group.networking.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.1.1.0/24"]

  delegation {
    name = "delegation"
  
    service_delegation {
    name    = "Microsoft.Web/hostingEnvironments"
    actions = ["Microsoft.Network/virtualNetworks/subnets/action"]
    }
  }
}

# Route tables
resource "azurerm_route_table" "local" {
  name                          = "local"
  location            = azurerm_resource_group.networking.location
  resource_group_name = azurerm_resource_group.networking.name
  disable_bgp_route_propagation = false

  route {
    name           = "route1"
    address_prefix = "10.1.0.0/16"
    next_hop_type  = "vnetlocal"
  }

  tags = {
    environment = "uat"
  }
}

resource "azurerm_subnet_route_table_association" "frontend" {
  subnet_id      = azurerm_subnet.frontend.id
  route_table_id = azurerm_route_table.local.id
}

# Frontend App Security
resource "azurerm_network_security_group" "frontend" {
  name                = "${local.prefix}-frontend-sg"
  location            = azurerm_resource_group.frontend.location
  resource_group_name = azurerm_resource_group.frontend.name

  security_rule {
    name                       = "test123"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "*"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }
}

resource "azurerm_subnet_network_security_group_association" "frontend" {
  subnet_id                 = azurerm_subnet.frontend.id
  network_security_group_id = azurerm_network_security_group.frontend.id
}

[So what I am trying to do above is to make sure that App Service is in the private subnet (with no access from internet) and controlled by Security Group rule. I am coming from AWS world and this is the most logical setup to achieve it for me. Please correct me if I'm wrong again.]

Now, I want to have internet-facing Load Balancer (or Application Gateway) to gather the traffic based on the URL path or port, and then to route it to the correct App Service (which itself is in the private subnet). And I have no idea what's the best practice there.

This is the piece of code I have at the moment:

# Public IP
resource "azurerm_public_ip" "agw" {
  name                = "${local.prefix}-agw-pip"
  location              = azurerm_resource_group.agw.location
  resource_group_name   = azurerm_resource_group.agw.name
  allocation_method   = "Static"
  sku                 = "Standard"
  tags                = azurerm_resource_group.agw.tags
}

# Application Gateway
resource "azurerm_application_gateway" "agw" {
  name                = "${local.prefix}-agw"
  location              = azurerm_resource_group.agw.location
  resource_group_name   = azurerm_resource_group.agw.name

  sku {
    name     = "WAF_Medium"
    tier     = "WAF"
    capacity = 2
  }

  waf_configuration {
    enabled          = "true"
    firewall_mode    = "Detection"
    rule_set_type    = "OWASP"
    rule_set_version = "3.0"
  }

  gateway_ip_configuration {
    name      = "subnet"
    subnet_id = <to be created>
  }

  frontend_port {
    name = "http"
    port = 80
  }

  frontend_ip_configuration {
    name                 = "frontend"
    public_ip_address_id = "${azurerm_public_ip.agw.id}"
  }

  backend_address_pool {
    name        = "AppService"
    "fqdn_list" = ["${azurerm_app_service.frontend.name}.azurewebsites.net"]
  }

  http_listener {
    name                           = "http"
    frontend_ip_configuration_name = "frontend"
    frontend_port_name             = "http"
    protocol                       = "Http"
  }

  probe {
    name                = "probe"
    protocol            = "http"
    path                = "/"
    host                = "${azurerm_app_service.frontend.name}.azurewebsites.net"
    interval            = "30"
    timeout             = "30"
    unhealthy_threshold = "3"
  }

  backend_http_settings {
    name                  = "http"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
    probe_name            = "probe"
  }

  request_routing_rule {
    name                       = "http"
    rule_type                  = "Basic"
    http_listener_name         = "http"
    backend_address_pool_name  = "AppService"
    backend_http_settings_name = "http"
  }
}

But I have no idea how to "connect" this Application Gateway with the App Services, and if it's all right to have it along with App Service Environment.

azure-application-gateway terraform azure-web-app-service azure
1 Answers

Your script seems to be fine except for

  1. backend_address_pool {
    name        = "AppService"
    "fqdn_list" = ["${azurerm_app_service.frontend.name}.azurewebsites.net"]
  }

    "fqdn_list" is not used now . it has been replaced by fqdns. So, you can use the below:

      backend_address_pool {
    name        = "AppService"
    fqdns = ["${azurerm_app_service.frontend.name}.azurewebsites.net"]
  }

  2.  gateway_ip_configuration {
    name      = "subnet"
    subnet_id = <to be created>
  }

    In this you have to create a subnet which will be used by your application gateway in the same vnet . So, you can use :

      gateway_ip_configuration {
    name      = "subnet"
    subnet_id = azurerm_subnet.appgw.id
  }

  3. backend_http_settings {
    name                  = "http"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
    probe_name            = "probe"
  }

    Add a feature flag "PickHostNameFromBackendAddress: true":

      backend_http_settings {
    name                  = "http"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
    probe_name            = "probe"
    pick_host_name_from_backend_address = true
  }

So, after this script is applied it creates the resources and you can see the backend pool in application gateway for your app service will be healthy.

But a better solution for your requirement is App gateway with private endpoint:

You can use the code below:

    provider "azurerm" {
    features{}
}

data "azurerm_resource_group" "rg" {
  name     = "yourresourcegroup"
}

# Virtual Network
resource "azurerm_virtual_network" "vnet" {
  name                = "ansumanapp-vnet"
  location            = data.azurerm_resource_group.rg.location
  resource_group_name = data.azurerm_resource_group.rg.name
  address_space       = ["10.4.0.0/16"]
}

# Subnets for App Service instances and app gateway
resource "azurerm_subnet" "appserv" {
  name                 = "frontend-app"
  resource_group_name  = data.azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.4.1.0/24"]
  enforce_private_link_endpoint_network_policies = true
  }

resource "azurerm_subnet" "appgw" {
  name                 = "appgw-subnet"
  resource_group_name  = data.azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.4.2.0/24"]
  enforce_private_link_endpoint_network_policies = true
  }

 
# App Service Plan
resource "azurerm_app_service_plan" "frontend" {
  name                = "ansuman-frontend-asp"
  location            = data.azurerm_resource_group.rg.location
  resource_group_name = data.azurerm_resource_group.rg.name
  kind                = "Linux"
  reserved            = true

  sku {
    tier = "Premium"
    size = "P1V2"
  }
}


# Main App Service
resource "azurerm_app_service" "frontend" {
  name                = "ansuman-frontend-app"
  location            = data.azurerm_resource_group.rg.location
  resource_group_name = data.azurerm_resource_group.rg.name
  app_service_plan_id = azurerm_app_service_plan.frontend.id

}
#private endpoint

resource "azurerm_private_endpoint" "example" {
  name                = "${azurerm_app_service.frontend.name}-endpoint"
  location            = data.azurerm_resource_group.rg.location
  resource_group_name = data.azurerm_resource_group.rg.name
  subnet_id           = azurerm_subnet.appserv.id
  

  private_service_connection {
    name                           = "${azurerm_app_service.frontend.name}-privateconnection"
    private_connection_resource_id = azurerm_app_service.frontend.id
    subresource_names = ["sites"]
    is_manual_connection = false
  }
}

# private DNS
resource "azurerm_private_dns_zone" "example" {
  name                = "privatelink.azurewebsites.net"
  resource_group_name = data.azurerm_resource_group.rg.name
}

#private DNS Link
resource "azurerm_private_dns_zone_virtual_network_link" "example" {
  name                  = "${azurerm_app_service.frontend.name}-dnslink"
  resource_group_name   = data.azurerm_resource_group.rg.name
  private_dns_zone_name = azurerm_private_dns_zone.example.name
  virtual_network_id    = azurerm_virtual_network.vnet.id
  registration_enabled = false
}

resource "azurerm_public_ip" "agw" {
  name                = "ansuman-agw-pip"
  location              = data.azurerm_resource_group.rg.location
  resource_group_name   = data.azurerm_resource_group.rg.name
  allocation_method   = "Dynamic"
  sku                 = "Basic"
}

# Application Gateway
resource "azurerm_application_gateway" "agw" {
  name                = "ansuman-agw"
  location              = data.azurerm_resource_group.rg.location
  resource_group_name   = data.azurerm_resource_group.rg.name

  sku {
    name     = "WAF_Medium"
    tier     = "WAF"
    capacity = 2
  }

  waf_configuration {
    enabled          = "true"
    firewall_mode    = "Detection"
    rule_set_type    = "OWASP"
    rule_set_version = "3.0"
  }

  gateway_ip_configuration {
    name      = "subnet"
    subnet_id = azurerm_subnet.appgw.id
  }

  frontend_port {
    name = "http"
    port = 80
  }

  frontend_ip_configuration {
    name                 = "frontend"
    public_ip_address_id = "${azurerm_public_ip.agw.id}"
  }

  backend_address_pool {
    name        = "AppService"
    fqdns = ["${azurerm_app_service.frontend.name}.azurewebsites.net"]
  }

  http_listener {
    name                           = "http"
    frontend_ip_configuration_name = "frontend"
    frontend_port_name             = "http"
    protocol                       = "Http"
  }

  probe {
    name                = "probe"
    protocol            = "http"
    path                = "/"
    host                = "${azurerm_app_service.frontend.name}.azurewebsites.net"
    interval            = "30"
    timeout             = "30"
    unhealthy_threshold = "3"
  }

  backend_http_settings {
    name                  = "http"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
    probe_name            = "probe"
    pick_host_name_from_backend_address = true
  }

  request_routing_rule {
    name                       = "http"
    rule_type                  = "Basic"
    http_listener_name         = "http"
    backend_address_pool_name  = "AppService"
    backend_http_settings_name = "http"

  }
}

Note: App service Sku should be Premium and tier can be as per your requirement for it to have access with private-endpoint.

Reference: Securing Azure Web Apps using Application Gateways and vNets

user_159692990
Related