Terraform: retrieve the nginx ingress controller Load Balancer IP

user_4504729 0
Viewed 963

I'm trying to get the nginx ingress controller load balancer ip in Azure AKS. I figured I would use the kubernetes provider via:

data "kubernetes_service" "nginx_service" {
  metadata {
    name      = "${local.ingress_name}-ingress-nginx-controller"
    namespace = local.ingress_ns
  }

  depends_on = [helm_release.ingress]
}

However, i'm not seeing the IP address, this is what i get back:

nginx_service = [
      + {
          + cluster_ip                  = "10.0.165.249"
          + external_ips                = []
          + external_name               = ""
          + external_traffic_policy     = "Local"
          + health_check_node_port      = 31089
          + load_balancer_ip            = ""
          + load_balancer_source_ranges = []
          + port                        = [
              + {
                  + name        = "http"
                  + node_port   = 30784
                  + port        = 80
                  + protocol    = "TCP"
                  + target_port = "http"
                },
              + {
                  + name        = "https"
                  + node_port   = 32337
                  + port        = 443
                  + protocol    = "TCP"
                  + target_port = "https"
                },
            ]
          + publish_not_ready_addresses = false
          + selector                    = {
              + "app.kubernetes.io/component" = "controller"
              + "app.kubernetes.io/instance"  = "nginx-ingress-internal"
              + "app.kubernetes.io/name"      = "ingress-nginx"
            }
          + session_affinity            = "None"
          + type                        = "LoadBalancer"
        },
   ]

However when I pull down the service via kubectl I can get the IP address via:

 kubectl get svc nginx-ingress-internal-ingress-nginx-controller -n nginx-ingress -o json | jq -r '.status.loadBalancer.ingress[].ip'
10.141.100.158

Is this a limitation of kubernetes provider for AKS? If so, what is a workaround other people have used? My end goals is to use the IP to configure the application gateway backend.

I guess I can use local-exec, but that seem hacky. Howerver, this might be my only option at the moment.

Thanks,

Jerry

azure-aks terraform kubernetes
2 Answers

although i strongly advise against creating resources inside Kubernetes with Terraform, you can do that:

Create a Public IP with Terraform -> Create the ingress-nginx inside Kubernetes with Terraform and pass annotations and loadBalancerIPwith data from your Terraform resources. The final manifest should look like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup
  name: ingress-nginx-controller
spec:
  loadBalancerIP: <YOUR_STATIC_IP>
  type: LoadBalancer

Terraform could look like this:

resource "kubernetes_service" "ingress_nginx" {
  metadata {
    name = "tingress-nginx-controller"
    
    annotations {
      "service.beta.kubernetes.io/azure-load-balancer-resource-group" = "${azurerm_resource_group.YOUR_RG.name}"
    }

  spec {
    selector = {
      app = <PLACEHOLDER>
    }
    port {
      port        = <PLACEHOLDER>
      target_port = <PLACEHOLDER>
    }

    type = "LoadBalancer"
    load_balancer_ip = "${azurerm_public_ip.YOUR_IP.ip_address}"
  }
}
user_167764510

Unfortunately, this is for internal ingress and not public facing and the IP is allocated dynamically. We currently dont want to use static ips

This is what I came up with:

module "load_balancer_ip" {
  count = local.create_ingress ? 1 : 0

  source  = "github.com/matti/terraform-shell-resource?ref=v1.5.0"
  command = "./scripts/get_load_balancer_ip.sh"

  environment = {
    KUBECONFIG = base64encode(module.aks.kube_admin_config_raw)
  }

  depends_on = [local_file.load_balancer_ip_script]
}

resource "local_file" "load_balancer_ip_script" {
  count = local.create_ingress ? 1 : 0

  filename = "./scripts/get_load_balancer_ip.sh"
  content  = <<-EOT
    #!/bin/bash
    echo $KUBECONFIG | base64 --decode > kubeconfig
    kubectl get svc -n ${local.ingress_ns} ${local.ingress_name}-ingress-nginx-controller --kubeconfig kubeconfig -o=jsonpath='{.status.loadBalancer.ingress[0].ip}'
    rm -f kubeconfig 2>&1 >/dev/null
  EOT
}

output nginx_ip {
  description = "IP address of the internal nginx controller"
  value = local.create_ingress ? module.load_balancer_ip[0].content : null
}
user_45047290
Related