nginx-ingress on GKE fails to route to path for configured service

user_1245756 0
Viewed 496

I am trying to configure an nginx ingress for a GKE cluster and define a path on a configured subdomain. It seems that even if I am able to successfully ping the host, and the domain binding is done correctly, I keep getting a 404 back whenever I try to access the configured path.

My goal is to be able to have a single static IP configured for my ingress controller and expose multiple services on different paths.

Below you can find my deployment files - one more thing that I would add is that I am using Terraform to automate the configuration and deployment of GCP and Kubernetes resources.

After the GKE cluster is successfully provisioned, I first deploy the official nginx-ingress controller from here - below my Terraform script that configures and deploys the controller with a custom static IP that I provisioned on GCP.

resource "helm_release" "nginx" {
  name  = "nginx"
  chart = "nginx-stable/nginx-ingress"
  timeout = 900

  set {
    name = "controller.stats.enabled"
    value = true
  }

  set {
    name  = "controller.service.type"
    value = "LoadBalancer"
  }

  set {
    name  = "controller.service.loadBalancerIP"
    value = "<MY_STATIC_IP_ADDRESS>"
  }
}

Below my ingress configuration that I also deploy via Terraform:

resource "kubernetes_ingress" "ingress" {
  wait_for_load_balancer = true

  metadata {
    name = "app-ingress"

    annotations = {
        "kubernetes.io/ingress.class": "nginx"
        "nginx.ingress.kubernetes.io/rewrite-target": "/"
        "kubernetes.io/ingress.global-static-ip-name": <MY_STATIC_IP_ADDRESS>
    }
  }

  spec {
    rule {
      host = custom.my_domain.com
      http {
        path {
          backend {
            service_name = "app-service"
            service_port = 5000
          }

          path = "/app"
        }
      }
    }
  }
}

And the resulting ingress configuration as taken from GCP:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.global-static-ip-name: static-ip-name
    nginx.ingress.kubernetes.io/rewrite-target: /
  creationTimestamp: "2021-04-14T20:28:41Z"
  generation: 7
  name: app-ingress
  namespace: default
  resourceVersion: HIDDEN
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
  uid: HIDDEN
spec:
  rules:
  - host: custom.my_domain.com
    http:
      paths:
      - backend:
          serviceName: app-service
          servicePort: 5000
        path: /app
status:
  loadBalancer:
    ingress:
    - ip: <MY_STATIC_IP_ADDRESS>

And the output for the kubectl describe ingress app-ingress command:

Name:             app-ingress
Namespace:        default
Address:          <MY_STATIC_IP_ADDRESS>
Default backend:  default-http-backend:80 (192.168.10.8:8080)
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  custom.my_domain.com
                        /app   app-service:5000 (192.168.10.11:5000)
Annotations:            kubernetes.io/ingress.class: nginx
                        kubernetes.io/ingress.global-static-ip-name: static-ip-name
                        nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason          Age                From                      Message
  ----    ------          ----               ----                      -------
  Normal  AddedOrUpdated  16m (x6 over 32m)  nginx-ingress-controller  Configuration for default/app-ingress was added or updated

I deployed the application that I am trying to expose by using the following configuration files:

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: default

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  type: NodePort
  ports:
    - port: 5000
      targetPort: 5000
      protocol: TCP
      name: http

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  replicas: 1
  template:
    spec:
      restartPolicy: Always
      volumes:
        - name: app-pvc
          persistentVolumeClaim:
            claimName: app-pvc
      containers:
      - name: app-container
        image: "eu.gcr.io/<PROJECT_ID>/<IMAGE_NAME>:VERSION_TAG"
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 5000
        livenessProbe:
          tcpSocket:
            port: 5000
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 5000
          initialDelaySeconds: 15
          periodSeconds: 20
        volumeMounts:
          - mountPath: "/data"
            name: app-pvc

Everything gets deployed successfully, as I am able to directly connect to the application locally via the configured service by running the following command:

kubectl port-forward service/app-service 5000:5000

This allows me to access the application in my browser and everything works as intended.

To make sure that <MY_STATIC_IP_ADDRESS> that I configured is properly bound to custom.my_domain.com, I tried to ping the host and I do get the right response back:

ping custom.my_domain.com

Pinging custom.my_domain.com [<MY_STATIC_IP_ADDRESS>] with 32 bytes of data:
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=36ms TTL=113
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=37ms TTL=113
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=36ms TTL=113
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=45ms TTL=113

Ping statistics for <MY_STATIC_IP_ADDRESS>:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 36ms, Maximum = 45ms, Average = 38ms

Even if everything appears to be working as intended, whenever I try to navigate to custom.my_domain.com/app in my browser, I keep getting the following response in my browser, even after waiting for more than 30m to make sure that the ingress configuration has been properly registered on GCP: enter image description here

And this is the entry that shows up in the logs of my nginx-controller pod:

<HIDDEN_LOCAL_IP> - - [14/Apr/2021:21:18:10 +0000] "GET /app/ HTTP/1.1" 404 232 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0" "-"

UPDATE #1

It appears that if I update my ingress to directly expose the targeted service on the / path, it works as intended. Below the updated configuration. Still, it appears that if I try to set any other path, it does not work.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.global-static-ip-name: static-ip-name
    nginx.ingress.kubernetes.io/rewrite-target: /
  creationTimestamp: "2021-04-14T20:28:41Z"
  generation: 7
  name: app-ingress
  namespace: default
  resourceVersion: HIDDEN
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
  uid: HIDDEN
spec:
  rules:
  - host: custom.my_domain.com
    http:
      paths:
      - backend:
          serviceName: app-service
          servicePort: 5000
        path: /
status:
  loadBalancer:
    ingress:
    - ip: <MY_STATIC_IP_ADDRESS>

Update #2

After going through the materials shared by @jccampanero in the comments section, I was able to get a working configuration.

Instead of using nginx-stable which is referenced on the official nginx website, I used the one here and updated my Terraform script accordingly to use this one with the exact same configuration I had.

Afterwards, I had to update my ingress by following the documentation here - below the updated configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.global-static-ip-name: static-ip-name
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  creationTimestamp: "2021-04-14T20:28:41Z"
  generation: 7
  name: app-ingress
  namespace: default
  resourceVersion: HIDDEN
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
  uid: HIDDEN
spec:
  rules:
  - host: custom.my_domain.com
    http:
      paths:
      - backend:
          serviceName: app-service
          servicePort: 5000
        path: /app(/|$)(.*)
status:
  loadBalancer:
    ingress:
    - ip: <MY_STATIC_IP_ADDRESS>
nginx-ingress kubernetes google-cloud-platform nginx networking
1 Answers

As indicated in the question comments and in the question itself, very well documented by @vladzam, two are the reasons of the problem.

On one hand, the nginx ingress controller available through the Helm stable channel seems to be deprecated in favor of the new ingress-nginx controller - please, see the Github repo and the official documentation.

On the other, it seems to be a problem related to the definition of the Rewrite target annotation. According to the docs:

Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group.

As a consequence, it is necessary to modify the definition of the ingress resource to take into account this change. For instance:

$ echo '
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: rewrite
  namespace: default
spec:
  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)
' | kubectl create -f -

The question itself provides the exact ingress resource definition.

user_139424480
Related