Kubernetes Graceful Shutdown: Continue to serve traffic during termination

user_6722596 0
Viewed 352

This question is about graceful shutdowns in Kubernetes.

I have a set of servers that are eventually consistent and requires each of them to communicate with each other constantly, even if some servers are terminating.

Since the servers communicate via HTTP requests, I need to ensure that the terminating server needs to be able to continue to receive HTTP requests in order to pass on relevant information to its peers.

Based on my understanding, there will be a few things that will happen in parallel when a server is shutting down: it will be removed from endpoints list for service and that if there is a preStop hook it will be run. Originally I thought if I use a preStop handler and perform a sleep command then I can let the server continue to server traffic. But after some experimentation I realized that the server will stop serving traffic even before the sleep ends.

I was wondering if there are any ways I can configure the servers to keep receiving traffic while it's shutting down.

kubernetes server yaml
1 Answers

Is there a way to still be able to reach the server while it's gracefully shutting down?

Yes, configure your Service with:

spec.publishNotReadyAddresses: true

Then your pods will receive traffic even though they are in a Terminating state, as the Kubernetes docs explain:

publishNotReadyAddresses, when set to true, indicates that DNS implementations must publish the notReadyAddresses of subsets for the Endpoints associated with the Service. The default value is false. The primary use case for setting this field is to use a StatefulSet's Headless Service to propagate SRV records for its Pods without respect to their readiness for purpose of peer discovery.

Note that spec.publishNotReadyAddresses only works in Kubernetes v1.11+. In older clusters, use the service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" annotation to get a similar behaviour.

user_38656260
Related