Kubernetes GKE 502 server error failed to pick backend when reloaded multiple times

Usama Baig 0
Viewed 56

I have a vue js website deployed on Google Kubernetes Engine. Below is its deployment code:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prod-e4o
  namespace: website
  labels:
    app: prod-e4o
spec:
  replicas: 1
  revisionHistoryLimit: 10
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app: prod-e4o
  template:
    metadata:
      labels:
        app: prod-e4o
    spec:
      containers:
        - image: gcr.io/*******
          name: app
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 500m
              ephemeral-storage: 1Gi
              memory: 0.75Gi
            requests:
              cpu: 500m
              ephemeral-storage: 1Gi
              memory: 0.75Gi
          ports:
            - containerPort: 3002
              name: http
          lifecycle:
            preStop:
              exec:
                command: [ "/bin/sh", "-c", "sleep 45" ]
          livenessProbe:
            initialDelaySeconds: 45
            timeoutSeconds: 5
            periodSeconds: 30
            tcpSocket:
              port: 3002
          readinessProbe:
            initialDelaySeconds: 45
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3002

Problem

The issue is when I reload the website multiple times quickly, it suddenly starts displaying 502 server error page for about 5 to 20 seconds (sometimes the error is gone in 5 seconds, other times it takes 20 seconds to become available). Here's the exact error message.

Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.

Things I have tried so far:

I have came across some questions with the same problems where it was suggested this issue might be due to resources. I have tried by increasing resources and making replicas 2 but the issue exist.

Some suggestions were that this is due to load balancer. I suspect that too because this is the log generated in Google cloud when this issue comes but I am not sure how this can be fixed:

{
  "insertId": "1ulcvwsf64b19x",
  "jsonPayload": {
    "remoteIp": "****",
    "statusDetails": "failed_to_pick_backend",
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
  },
  "httpRequest": {
    "requestMethod": "GET",
    "requestUrl": "****",
    "requestSize": "278",
    "status": 502,
    "responseSize": "545",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0",
    "remoteIp": "63.33.68.103",
    "referer": "www.wolframalpha.com",
    "latency": "0.018271s"
  },
  "resource": {
    "type": "http_load_balancer",
    "labels": {
      "zone": "global",
      "url_map_name": "k8s2-um-97ve13di-website-website-c2lsezd7",
      "forwarding_rule_name": "k8s2-fs-97ve13di-website-website-c2lsezd7",
      "backend_service_name": "k8s1-d8188420-website-prod-e4o-3002-132d57a2",
      "target_proxy_name": "k8s2-ts-97ve13di-website-website-c2lsezd7",
      "project_id": "****"
    }
  },
  "timestamp": "2022-09-09T16:39:41.867486Z",
  "severity": "WARNING",
  "logName": "projects/****/logs/requests",
  "trace": "projects/****/traces/58bf40ba3fc95346005510c395c13f97",
  "receiveTimestamp": "2022-09-09T16:39:47.062198390Z",
  "spanId": "98c531a52c84ccca"
}

EDIT: As asked in the comment, the result of command kubectl describe <pod-name> -n <namespace> is below:

Name:         prod-**-****-t4gz7
Namespace:    website
Priority:     0
Node:         gk3-prod-cluster-nap-zrypjsnr-e8f03002-qfsc/10.164.15.225
Start Time:   Thu, 08 Sep 2022 22:17:30 +0500
Labels:       app=prod-e4o
              pod-template-hash=785d68c9d6
Annotations:  kubectl.kubernetes.io/restartedAt: 2022-09-08T17:17:30Z
              seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:       Running
IP:           10.121.0.172
IPs:
  IP:           10.121.0.172
Controlled By:  ReplicaSet/prod-**-**
Containers:
  app:
    Container ID:   containerd://4da23c31271567c9237649e692a50001794eb16f61b005fb32d63413dd2cecd5
    Image:          ****
    Image ID:       ****
    Port:           3002/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 08 Sep 2022 22:18:24 +0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                500m
      ephemeral-storage:  1Gi
      memory:             768Mi
    Requests:
      cpu:                500m
      ephemeral-storage:  1Gi
      memory:             768Mi
    Liveness:             tcp-socket :3002 delay=45s timeout=5s period=30s #success=1 #failure=3
    Readiness:            tcp-socket :3002 delay=45s timeout=1s period=10s #success=1 #failure=3
    Environment:          <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tgh9v (ro)
Readiness Gates:
  Type                                       Status
  cloud.google.com/load-balancer-neg-ready   True 
Conditions:
  Type                                       Status
  cloud.google.com/load-balancer-neg-ready   True 
  Initialized                                True 
  Ready                                      True 
  ContainersReady                            True 
  PodScheduled                               True 
Volumes:
  kube-api-access-tgh9v:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Output for the command kubectl describe ingress --namespace website

Name:             website
Labels:           <none>
Namespace:        website
Address:          34.102.204.15
Default backend:  default-http-backend:80 (10.121.0.75:8080)
Rules:
  Host                           Path  Backends
  ----                           ----  --------
  
  prod-url.com              
                                    prod-***:http (10.121.0.172:3002)
  www.prod-url.com          
                                    prod-***:http (10.121.0.172:3002)
  beta-url.com         
                                    beta-***:http (10.121.0.164:3002)

Annotations:                     ingress.gcp.kubernetes.io/pre-shared-cert: tls
                                 ingress.kubernetes.io/backends:
                                   {"k8s-be-31929--d81884208ac05b04":"HEALTHY","k8s1-d8188420-website-beta-cw-3001-74d92dad":"HEALTHY","k8s1-d8188420-website-beta-e4o-3002-0...
                                 ingress.kubernetes.io/forwarding-rule: k8s2-fr-97ve13di-website-website-c2lsezd7
                                 ingress.kubernetes.io/https-forwarding-rule: k8s2-fs-97ve13di-website-website-c2lsezd7
                                 ingress.kubernetes.io/https-target-proxy: k8s2-ts-97ve13di-website-website-c2lsezd7
                                 ingress.kubernetes.io/redirect-url-map: k8s2-rm-97ve13di-website-website-c2lsezd7
                                 ingress.kubernetes.io/ssl-cert: capwell-tls,expert4office-tls
                                 ingress.kubernetes.io/static-ip: k8s2-fr-97ve13di-website-website-c2lsezd7
                                 ingress.kubernetes.io/target-proxy: k8s2-tp-97ve13di-website-website-c2lsezd7
                                 ingress.kubernetes.io/url-map: k8s2-um-97ve13di-website-website-c2lsezd7
                                 kubernetes.io/ingress.allow-http: true
                                 networking.gke.io/v1beta1.FrontendConfig: redirect-http-to-https
Events:
  Type    Reason  Age                       From                     Message
  ----    ------  ----                      ----                     -------
  Normal  Sync    9m34s (x1024 over 6d14h)  loadbalancer-controller  Scheduled for sync

Output of the command kubectl describe svc --namespace website

Name:              prod-***
Namespace:         website
Labels:            app=prod-***
Annotations:       cloud.google.com/neg: {"ingress":true}
                   cloud.google.com/neg-status:
                     {"network_endpoint_groups":{"3002":"k8s1-d8188420-website-prod-***-3002-132d57a2"},"zones":["europe-west4-a","europe-west4-b"]}
Selector:          app=prod-***
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.121.128.147
IPs:               10.121.128.147
Port:              http  3002/TCP
TargetPort:        http/TCP
Endpoints:         10.121.0.172:3002
Session Affinity:  None
Events:            <none>

Output of the command kubectl describe ep -n website prod-e4o

Name:         prod-e4o
Namespace:    website
Labels:       app=prod-e4o
Annotations:  <none>
Subsets:
  Addresses:          10.121.0.172
  NotReadyAddresses:  <none>
  Ports:
    Name  Port  Protocol
    ----  ----  --------
    http  3002  TCP

Events:  <none>
google-kubernetes-engine kubernetes
1 Answers

From the details that you have provided above I can see that from the logs generated there is a "statusDetails": "failed_to_pick_backend", this could mean that there is unhealthy backend. You can configure the correct firewall rules for health checks.

As you have also mention that it is displaying the 502 server error page, if this error 502 persist after you complete the load balancer configuration, it's likely that either:

You can refer also to this documentation on Troubleshooting general connectivity issues.

Jon B.0
Related