AppGateway / AKS / AGIC / Let's Encrypt Not Working - Secret Not Found

user_16309039 0
Viewed 1817

Describe the bug Followed doco here, but it's out of date so had to guess ... https://docs.microsoft.com/en-us/azure/application-gateway/ingress-controller-letsencrypt-certificate-application-gateway. When applying the manifest it only creates an http listerner and not https. It isn't creating the cert, and erroring with 'Secret Not Found'.

agic = mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.4.0
cert-manager = quay.io/jetstack/cert-manager-controller:v1.4.3
aks kubernetes = 1.20.7

To Reproduce See yaml below. This works fine if I tweak to use a manually created secret / cert. When I try to create via letsencrypt I get a 'SecretNotFound' error on the AGIC pod.

Ingress Controller details

  • Output of kubectl describe pod <ingress controller>.
Name:         ingress-appgw-deployment-9ffdc54cb-629hg
Namespace:    kube-system
Priority:     0
Node:         aks-default-32636497-vmss000000/10.94.112.4
Start Time:   Wed, 18 Aug 2021 09:59:16 +0100
Labels:       app=ingress-appgw
              kubernetes.azure.com/managedby=aks
              pod-template-hash=9ffdc54cb
Annotations:  checksum/config: 78a4d434072823accba40908961d40922d59acb0000a42182add8d60cde0c9a1
              cluster-autoscaler.kubernetes.io/safe-to-evict: true
              kubernetes.azure.com/metrics-scrape: true
              prometheus.io/path: /metrics
              prometheus.io/port: 8123
              prometheus.io/scrape: true
              resource-id:
                /subscriptions/2bc7b65e-18d6-42ae-afb2-e66d50be6b05/resourceGroups/rg-prd-agwaks-210818-0950/providers/Microsoft.ContainerService/managedC...
Status:       Running
IP:           10.94.112.10
IPs:
  IP:           10.94.112.10
Controlled By:  ReplicaSet/ingress-appgw-deployment-9ffdc54cb
Containers:
  ingress-appgw-container:
    Container ID:   containerd://93e66897c6646d7f6efbf9496646633f13424917a183e85790df0e6c17cc7a91
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.4.0
    Image ID:       sha256:533f2cbe57fa92d27be5939f8ef8dc50537d6e1240502c8c727ac4020545dd34
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 18 Aug 2021 09:59:18 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     700m
      memory:  100Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-appgw-cm  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/kubernetes/azure.json
      AGIC_POD_NAME:                  ingress-appgw-deployment-9ffdc54cb-629hg (v1:metadata.name)
      AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
      KUBERNETES_PORT_443_TCP_ADDR:   aks-prd-agwaks-210818-0950-dns-37f5d052.hcp.northeurope.azmk8s.io
      KUBERNETES_PORT:                tcp://aks-prd-agwaks-210818-0950-dns-37f5d052.hcp.northeurope.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:        tcp://aks-prd-agwaks-210818-0950-dns-37f5d052.hcp.northeurope.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:        aks-prd-agwaks-210818-0950-dns-37f5d052.hcp.northeurope.azmk8s.io
    Mounts:
      /etc/kubernetes/azure.json from cloud-provider-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-appgw-sa-token-cdmtp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  cloud-provider-config:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  ingress-appgw-sa-token-cdmtp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-appgw-sa-token-cdmtp
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>
  • Output of kubectl logs .
I0818 19:43:07.518122       1 configbuilder.go:221] Invalid custom port configuration (0). Setting listener port to default : 80
I0818 19:43:07.518180       1 requestroutingrules.go:111] Bound basic rule: rr-12754dc8633d87433e25740857ea6708 to listener: fl-12754dc8633d87433e25740857ea6708 ([dev.rhod3rz.com    ], 80) for backend pool pool-default-aspnetapp-dev-80-bp-80 and backend http settings bp-default-aspnetapp-dev-80-80-aspnetapp-dev
I0818 19:43:07.518319       1 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"aspnetapp-dev", UID:"8086e92d-f9a4-4806-afd1-42c24f4f0722", APIVersion:"extensions/v1beta1", ResourceVersion:"90240", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [default/dev]
  • Manifest file.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: rhod3rz@outlook.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
    - http01:
        ingress:
          class: azure/application-gateway
---
apiVersion: v1
kind: Pod
metadata:
  name: aspnetapp-dev
  labels:
    app: aspnetapp-dev
spec:
  containers:
  - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
    name: aspnetapp-image
    ports:
    - containerPort: 80
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: aspnetapp-dev
spec:
  selector:
    app: aspnetapp-dev
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: aspnetapp-dev
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt-staging
    cert-manager.io/acme-challenge-type: http01
spec:
  tls:
    - hosts:
      - "dev.rhod3rz.com"
    - secretName: dev
  rules:
  - host: "dev.rhod3rz.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: aspnetapp-dev
            port:
              number: 80
  • kubectl describe ingress.
Events:
  Type     Reason          Age                From                       Message
  ----     ------          ----               ----                       -------
  Warning  BadConfig       40m (x2 over 40m)  cert-manager               TLS entry 0 is invalid: TLS entry for hosts [dev.rhod3rz.com] must specify a secretName
  Warning  BadConfig       40m (x2 over 40m)  cert-manager               TLS entry 1 is invalid: secret "dev" for ingress TLS has no hosts specified
  Warning  SecretNotFound  40m (x2 over 40m)  azure/application-gateway  Unable to find the secret associated to secretId: [default/dev]
kubernetes azure ssl
2 Answers

If you are using the cluster issuer with the ingress you have to pass the value of

privateKeySecretRef:
      name: example-issuer-account-key

inside the ingress only as a secret.

If you will check using command

kubectl get secret

you will see the secret inside a namespace with name : example-issuer-account-key

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: rhod3rz@outlook.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
    - http01:
        ingress:
          class: azure/application-gateway

ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: aspnetapp-dev
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt-staging
    cert-manager.io/acme-challenge-type: http01
spec:
  tls:
    - hosts:
      - "dev.rhod3rz.com"
    - secretName: example-issuer-account-key
  rules:
  - host: "dev.rhod3rz.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: aspnetapp-dev
            port:
              number: 80

also note that you are using staging certificate from let's encrypt so could be possible you will see SSL error into browser as it's staging certificate.

For production use case, you have to change server in clusterissuer.

Staging : https://acme-staging-v02.api.letsencrypt.org/directory

Production : https://acme-v02.api.letsencrypt.org/directory

user_55258240

Annoyingly it was just an extra hyphen '-' breaking things :-(

  • Snippet
# - secretName: banana # arghh ... the '-' was what was causing it to fail :-(
    secretName: banana # < cert-manager will store the created certificate in this secret.
  • Full working manifest.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-issuer
spec:
  acme:
    email: rhod3rz@outlook.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory  # < use this staging issuer when testing to avoid hitting rate limits on prod (50 per week).
    # server: https://acme-v02.api.letsencrypt.org/directory        # < use this prod issuer when ready to go live.
    privateKeySecretRef:
      name: apple
    solvers:
    - http01:
        ingress:
          class: azure/application-gateway
---
apiVersion: v1
kind: Pod
metadata:
  name: aspnetapp-dev
  labels:
    app: aspnetapp-dev
spec:
  containers:
  - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
    name: aspnetapp-image
    ports:
    - containerPort: 80
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: aspnetapp-dev
spec:
  selector:
    app: aspnetapp-dev
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
# https://cert-manager.io/docs/usage/ingress/
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: aspnetapp-dev
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway # < add annotation indicating the ingress to use.
    cert-manager.io/cluster-issuer: letsencrypt-issuer     # < add annotation indicating the cert issuer to use.
    appgw.ingress.kubernetes.io/ssl-redirect: "true"       # < add annotation to redirect 80 requests to 443.
    # cert-manager.io/acme-challenge-type: http01          # < this is no longer required; works without it.
spec:
  rules:
  - host: dev.rhod3rz.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: aspnetapp-dev
            port:
              number: 80
  tls: # < placing a host in the TLS config will determine what ends up in the cert's subjectAltNames.
  - hosts:
    - dev.rhod3rz.com
# - secretName: banana # arghh ... the '-' was what was causing it to fail :-(
    secretName: banana # < cert-manager will store the created certificate in this secret.
user_163090390
Related