K8s Pod Topology Spread is not respected after rollout?

Giving more visibility on the comment:

Daemonset worked and was easy enough. It won't work for our deployments with several pods per node, but there are mitigations there (descheduler) and it should self resolve as the cluster grows.

Please consider this answer as a workaround:

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

-- Kubernetes.io: Docs: Concepts: Workloads: Controllers: Daemonset

An example of it could be following:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx 
    spec:
      #nodeSelector: 
          #schedule: here 
      tolerations:
      # this toleration is to have the daemonset runnable on master nodes
      # remove it if your masters can't run pods
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: nginx
        image: nginx

This definition will spawn a Pod on each Node in the cluster. You can further limit the Pod scheduling by specifying a nodeSelector.

Assuming that you have some controller/logic responsible for tagging the nodes with a specific label you can schedule Pods on specific Nodes. The part responsible for it is commented out in above manifest:

      nodeSelector: 
          schedule: here 

The nodes (raven-sgdm and raven-xvvw are labeled): :

NAME         STATUS   ROLES    AGE    VERSION
raven-6k6m   Ready    <none>   159m   v1.20
raven-sgdm   Ready    <none>   159m   v1.20
raven-xvvw   Ready    <none>   159m   v1.20

The Daemonset:

NAME    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
nginx   2         2         2       2            2           schedule=here   99m

Additional resources:

• Kubernetes.io: Docs: Concepts: Workloads: Pods: Pod topology spread constraints
• Github.com: Kubernetes: Issues: 98215

At certain moment in time the skew will be proper.

But when the pods that are to be removed will be removed the skew might be skewed :)

Essentially you are facing the limitation described here:

Scaling down a Deployment may result in imbalanced Pods distribution.

One simple workaround I apply is scale it down + restart/deploy + scale it up.

Then the skew is perfectly fine!

kubectl rollout restart spins up new pods and then terminate old pods after all the new pods are up and running.

From the pod topology spread constraints known limitations section, constraints don't remain satisfied when pods are removed and the recommended mitigation is now to use Descheduler , which you already seemed to have been using from your comment.

In Kubernetes 1.25, you can now use the alpha feature matchLabelKeys to resolve this issue. Because there is an automatically generated pod-template-hash for each version of a deployment added to each pod, that and your app specific label can provide the hashed value to prevent mis-scheduling.

topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule
      matchLabelKeys:
        - app
        - pod-template-hash

matchLabelKeys is a list of pod label keys to select the pods over which spreading will be calculated. The keys are used to lookup values from the pod labels, those key-value labels are ANDed with labelSelector to select the group of existing pods over which spreading will be calculated for the incoming pod. Keys that don't exist in the pod labels will be ignored. A null or empty list means only match against the labelSelector.

With matchLabelKeys, users don't need to update the pod.spec between different revisions. The controller/operator just needs to set different values to the same label key for different revisions. The scheduler will assume the values automatically based on matchLabelKeys. For example, if users use Deployment, they can use the label keyed with pod-template-hash, which is added automatically by the Deployment controller, to distinguish between different revisions in a single Deployment.

Note: The matchLabelKeys field is an alpha field added in 1.25. You have to enable the MatchLabelKeysInPodTopologySpread feature gate in order to use it.

From the docs