failed_to_pick_backend from GCP Loadbalancer even healthy backend

user_15132444 0
Viewed 107

I am running a GKE Cluster with the GCP Load Balancer as my Ingress Controller.

However, I noticed that a specific request to the service hosted in this GKE Cluster was being rejected with and 502 error.

I checked the GCP Loadbalancer logs and I was able to see a return with statusDetails: "failed_to_pick_backend".

The health field is saying that my backend is healthy. Just to make sure I changed the health check type from HTTP to TCP to see if anything would change but it kept green.

So, what can I be missing if my GCP Loadbalancer is saying that my backend is reachable but at the same time it returns me an failed to pick backend message?

I really appreciate some help on that.

My ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    kubernetes.io/ingress.global-static-ip-name: my-app
    networking.gke.io/managed-certificates: my-app-managed-cert
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.allow-http: "false"
spec:
  defaultBackend:
    service:
      name: my-app-svc
      port:
        number: 3000

My service:

apiVersion: v1
kind: Service
metadata:
  name: my-app-svc
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000
  type: NodePort

Running kubectl describe pod <my-pod> I can see

Conditions:
  Type                                       Status
  cloud.google.com/load-balancer-neg-ready   True 
  Initialized                                True 
  Ready                                      True 
  ContainersReady                            True 
  PodScheduled                               True
google-kubernetes-engine kubernetes google-cloud-platform
1 Answers

Can you verify the timer of your health check? If this is set to 1 second or lower than the health checks that were set to higher value, failed_to_pick_backend error will normally occur.

I recommend you to change the timer to the default value 5 or higher and then test a new deployment. You can check more details about the health check timers on this link[1].

[1] https://cloud.google.com/load-balancing/docs/health-check-concepts

user_182411840
Related