I need help understanding how Kubernetes handles requests upon a pod restart.
If I have two pods running on a Kubernetes cluster and one dies or restarts midway, does the current request that the first pod is handling get transferred to the second pod or does the current request fail?
Kubernetes sends traffic only when the pod is running perfectly, now if in the mean time of serving the request pod dies then that traffic will fail.
In ideal scenario you would want to to have all client request handled properly without any broken connections. By itself, Kubernetes does not prevent you from this.
When your pod is started, its is being added as endpoint to all the services that label selector matches the labels that place on the pod. Pod needs to let Kubernetes know that it's ready by sending. Until this signal is being sent, it won't become service endpoint and won't receive any requests from you clients.
If you don't specify readiness probe for your pod, it is always considered ready and it will start receiving request almost immediately (as soon as kube-proxy will update the iptables rules). In this time if you application is not ready to serve you will receive connection refused errors. So having successfully probe return when you app is ready will make those error go away.
With pod deletion there are two things occurring in parallel, one being kubelet deleting the pod and the endpoint controller removing the service endpoint. In most causes the deletion of the pod is happening faster since endpoint removal chain is slightly longer. This will also means that upon deletion your pod might be deleted faster then the endpoint being removed. Delaying that removal until endpoint is removed will also help with refused connections.
