How to gracefully shutdown a Go service running on Kubernetes

user_2253858 0
Viewed 2393

I have an API written in Go that has been Dockerised and runs in a Kubernetes cluster on GKE.

At the moment my API server does not handle any shutdown scenarios such as a Pod dying or being purposefully brought down.

What set of UNIX signals should I expect to trap to gracefully shutdown the server and what circumstances would trigger them? For example, crashes, K8s shutdowns etc.

kubernetes docker go signals linux
2 Answers

Kubernetes sends a SIGTERM signal. So the graceful shutdown may look like this:

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
)

func main() {
    var srv http.Server

    idleConnsClosed := make(chan struct{})
    go func() {
        sigint := make(chan os.Signal, 1)

        // interrupt signal sent from terminal
        signal.Notify(sigint, os.Interrupt)
        // sigterm signal sent from kubernetes
        signal.Notify(sigint, syscall.SIGTERM)

        <-sigint

        // We received an interrupt signal, shut down.
        if err := srv.Shutdown(context.Background()); err != nil {
            // Error from closing listeners, or context timeout:
            log.Printf("HTTP server Shutdown: %v", err)
        }
        close(idleConnsClosed)
    }()

    if err := srv.ListenAndServe(); err != http.ErrServerClosed {
        // Error starting or closing listener:
        log.Printf("HTTP server ListenAndServe: %v", err)
    }

    <-idleConnsClosed
}

Also you should add Liveness and Readiness probes to your pods:

livenessProbe:
  httpGet:
    path: /health
    port: 80
readinessProbe:
  httpGet:
    path: /health
    port: 80
user_3502940

Answer https://stackoverflow.com/a/54239385/12641885 only partly correct. When Kubernetes send SIGTERM signal traffic can still route to the pod, because removing pod from endpoints set and SIGTERM signal is async - if pod is after SIGTERM endpoints may not have time to update. SIGTERM only politely asks to start wrapping up, but don't stop accepting new requests at once.

You can add sleep into your code after <-sigint or add into your Deployment

lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 10"]

More detailed info https://github.com/kubernetes/kubernetes/issues/86280#issuecomment-583173036

user_126418850
Related