Kubernetes deployment - control the maximum number of simultaneously terminating pods

user_2335775 0
Viewed 330

Using Kubernetes, we could control how many pods (app instances) are created with a new deployment rollout simultaneously. It's achievable by using properties max surge and max unavailable. If we have tens of app instances, the default configuration will rollout multiple new instances and at the same time, multiple instances will be terminating. The most preferable configuration for us is changing that to max surge = 1 and max unavailable = 0 (reason - to achieve smooth Kafka rebalancing). In that case, only a single app instance will be started at a specific time, and up to a few instances will be terminating.

As I see, the deployment strategy if we have max surge = N and max unavailable = 0 is the following:

  • step 1: N new pods (the first batch) starting
  • step 2: N pods terminating, and N new pods (the second batch) starting
  • step 3: N more pods terminating (and together with step 2, we could experience up to 2N terminating pods), and N new pods (the third batch) starting
  • step 4: same logic, and here we could experience more than 2N pods still terminating
  • and so on...

For our micro-service with total number of pods 30, with max surge = 1 and max unavailable = 0, we have up to 3 terminating pods simultaneously, and with max surge = 3 and max unavailable = 0 - up to 7 terminating pods.

Is it possible to control the maximum number of simultaneously terminating pods during deployment rollout? Let's say I want to see at most one pod in a terminating state. So until pod will not be shut down completely, no new pods started.

kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.10", GitCommit:"62876fc6d93e891aa7fbe19771e6a6c03773b0f7", GitTreeState:"clean", BuildDate:"2020-10-15T01:52:24Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.10", GitCommit:"62876fc6d93e891aa7fbe19771e6a6c03773b0f7", GitTreeState:"clean", BuildDate:"2020-10-15T01:43:56Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
kubernetes-deployment kubernetes-pod kubernetes deployment
1 Answers

maxUnavailable and maxSurge is definitely the way to go. The steps you listed are on point, and it is how it's supposed to work.

What you are experiencing is most probably a bug. There is one similiar issue on GitHub (#99513) about Kubernetes not respecting maxUnavailable and maxSurge.

If you can, create new issue on GitHub, as yours is a bit different then aforementioned one.

EDIT

I found another similiar issue (#95498) with a response, stating it's working as intended. [source]

When the Deployment controller performs a rolling update on a Deployment, it scales up/down ReplicaSets based on the .spec.replicas of the ReplicaSets it controls. When a ReplicaSet is scaled down, the ReplicaSet's .spec.replicas will be reduced and some of its Pods will be terminating, and the Deployment controller simply ignores those terminating Pods when calculating maxSurge / maxUnavailable.

Similarly, ReplicaSet controller ignores terminating Pods while doing operations, such as calculating its .status.availableReplicas

but I'm having my doubts. Creating new issue on GitHub, and getting a developer to look into it, is still the good option.

user_156593470
Related