GCP health check failing for kubernetes pod

Viewed 301

I'm trying to launch an application on GKE and the health checks made by the Ingress always fail.

Here's my full k8s yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tripvector
  labels:
    app: tripvector
spec:
  replicas: 1
  minReadySeconds: 60
  selector:
    matchLabels:
      app: tripvector
  template:
    metadata:
      labels:
        app: tripvector
    spec:

      containers:
      - name: tripvector
        readinessProbe:
          httpGet:
            port: 3000
            path: /healthz
          initialDelaySeconds: 30
          timeoutSeconds: 10
          periodSeconds: 11
        image: us-west1-docker.pkg.dev/triptastic-1542412229773/tripvector/tripvector:healthz2
        env:
          - name: ROOT_URL
            value: https://paymahn.tripvector.io/
          - name: MAIL_URL
            valueFrom:
              secretKeyRef:
                key: MAIL_URL
                name: startup
          - name: METEOR_SETTINGS
            valueFrom:
              secretKeyRef:
                key: METEOR_SETTINGS
                name: startup
          - name: MONGO_URL
            valueFrom:
              secretKeyRef:
                key: MONGO_URL
                name: startup
        ports:
        - containerPort: 3000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tripvector
spec:
  defaultBackend:
    service:
      name: tripvector-np
      port:
        number: 60000
---
apiVersion: v1
kind: Service
metadata:
  name: tripvector-np
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
spec:
  type: ClusterIP
  selector:
    app: tripvector
  ports:
    - protocol: TCP
      port: 60000
      targetPort: 3000

This yaml should do the following:

  1. make a deployment with my healthz2 image along with a readiness check at /healthz on port 3000 which is exposed by the image
  2. launch a cluster IP service
  3. launch an ingress

When I check for the status of the service I see it's unhealth:

 ❯❯❯ gcloud compute backend-services get-health k8s1-07274a01-default-tripvector-np-60000-a912870e --global
---
backend: https://www.googleapis.com/compute/v1/projects/triptastic-1542412229773/zones/us-central1-a/networkEndpointGroups/k8s1-07274a01-default-tripvector-np-60000-a912870e
status:
  healthStatus:
  - healthState: UNHEALTHY
    instance: https://www.googleapis.com/compute/v1/projects/triptastic-1542412229773/zones/us-central1-a/instances/gke-tripvector2-default-pool-78cf58d9-5dgs
    ipAddress: 10.12.0.29
    port: 3000
  kind: compute#backendServiceGroupHealth

It seems that the healthcheck is hitting the right port but this output doesn't confirm if it's hitting the right path. If I look up the health check object in the console I see the following:

enter image description here

Which confirms the GKE health check is hitting the healthz path.

I've verified in the following ways that the health check endpoint I'm using for the readiness probe works but something still isn't working properly:

  1. exec into the pod and run wget
  2. port forward the pod and check /healthz in my browser
  3. port forward the service and check /healthz in my browser

In all three instances above, I can see the /healthz endpoint working. I'll outline each one below.

Here's evidence that running wget from within the pod:

 ❯❯❯ k exec -it tripvector-65ff4c4dbb-vwvtr /bin/sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/tripvector # ls
bundle
/tripvector # wget localhost:3000/healthz
Connecting to localhost:3000 (127.0.0.1:3000)
saving to 'healthz'
healthz              100% |************************************************************************************************************************************************************|    25  0:00:00 ETA
'healthz' saved
/tripvector # cat healthz
[200] Healthcheck passed./tripvector #

Here's what happens when I perform a port forward from the pod to my local machine:

 ❯❯❯ k port-forward tripvector-65ff4c4dbb-vwvtr 8081:3000
Forwarding from 127.0.0.1:8081 -> 3000
Forwarding from [::1]:8081 -> 3000
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081

enter image description here

And here's what happens when I port forward from the Service object:

 2:53PM /Users/paymahn/code/tripvector/tripvector  ✘ 1 docker ⬆ ✱
 ❯❯❯ k port-forward svc/tripvector-np 8082:60000
Forwarding from 127.0.0.1:8082 -> 3000
Forwarding from [::1]:8082 -> 3000
Handling connection for 8082

enter image description here

How can I get the healthcheck for the ingress and network endpoint group to succeed so that I can access my pod from the internet?

0 Answers
Related