Why are my deployment/replicaset controllers not interfering with pods created with the same selectors?

Question

I am trying to test the theory of deployment controllers fighting over pods when creating two Deployments with exactly the same selector.

Here's my deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: busybox-container
        image: busybox
        command: ["/bin/sh"]
        args: ["-c", "sleep 10m"]
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
          - containerPort: 8080

deployment1.yaml only differs with metadata.name field having the value deployment-1 and spec.replicas being set to 1.

I create the deployments back to back as such:

❯ kubectl create -f deployment.yaml; kubectl create -f deployment1.yaml
deployment.apps/deployment created
deployment.apps/deployment-1 created

❯ kubectl get all
NAME                                READY   STATUS    RESTARTS   AGE
pod/deployment-1-7d95ff89fc-p6fg7   1/1     Running   0          18s
pod/deployment-7d95ff89fc-fvsvd     1/1     Running   0          18s
pod/deployment-7d95ff89fc-ktdq9     1/1     Running   0          18s
pod/deployment-7d95ff89fc-xnw42     1/1     Running   0          18s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/deployment     3/3     3            3           18s
deployment.apps/deployment-1   1/1     1            1           18s

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/deployment-1-7d95ff89fc   1         1         1       18s
replicaset.apps/deployment-7d95ff89fc     3         3         3       18s

I expect the second deployment to start interfering with the first deployment since the second deployment only wants 1 replica whereas the first deployment wants 3. I expect to see containers being deleted and recreated, but all is smooth sailing and I have 4 pods.

According to the Kubernetes documentation on a Deployment selector property, there should be interference.

If you have multiple controllers that have overlapping selectors, the controllers will fight with each other and won't behave correctly.

Now a similar question has been asked before and it names OwnerReferences as the reason for lack of interference, and the Owners and Dependents does state:

Owner references help different parts of Kubernetes avoid interfering with objects they don’t control.

Still the Owner/Dependent seems more concerned with garbage collection than with driving desired state. So then is the Deployment documentation incorrect/outdated? I thought a primary reason for selectors/labels was that controllers could pick up existing resources already in the system that were created outside of these higher resources.

I'm using the Kubernetes installation that comes installed w/ Docker Desktop for MacOS, version 1.22.4.

user_16739663 · Answer

I can confirm your observation. In current release there is no conflict within labels when you create two deployments with the same labels, however please keep in mind that:

Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods along with a lot of other useful features. [1]

A ReplicaSet is linked to its Pods via the Pods' metadata.ownerReferences field, which specifies what resource the current object is owned by. All Pods acquired by a ReplicaSet have their owning ReplicaSet's identifying information within their ownerReferences field. It's through this link that the ReplicaSet knows of the state of the Pods it is maintaining and plans accordingly.

A ReplicaSet identifies new Pods to acquire by using its selector. If there is a Pod that has no OwnerReference or the OwnerReference is not a Controller and it matches a ReplicaSet's selector, it will be immediately acquired by said ReplicaSet. [2]

Accordingly Replicaset is owned by Deployment Alternatives to ReplicaSet: Deployment (recommended)

Deployment is an object which can own ReplicaSets and update them and their Pods via declarative, server-side rolling updates. While ReplicaSets can be used independently, today they're mainly used by Deployments as a mechanism to orchestrate Pod creation, deletion and updates. When you use Deployments you don't have to worry about managing the ReplicaSets that they create. Deployments own and manage their ReplicaSets. As such, it is recommended to use Deployments when you want ReplicaSets. [3]

Note: Do not manage ReplicaSets owned by a Deployment. Consider opening an issue in the main Kubernetes repository if your use case is not covered below.

Going further: If you will inspect Replicaset you should notice Pod-template-hash label. Caution: Do not change this label.

The pod-template-hash label is added by the Deployment controller to every ReplicaSet that a Deployment creates or adopts.

This label ensures that child ReplicaSets of a Deployment do not overlap. It is generated by hashing the PodTemplate of the ReplicaSet and using the resulting hash as the label value that is added to the ReplicaSet selector, Pod template labels, and in any existing Pods that the ReplicaSet might have. [4]

I thought a partially reason for selectors/labels was that controllers could pick up existing resources already in the system that were created outside of these higher resources. This is basically true. As an example please take a look at: example in this doc.

Non-Template Pod acquisitions:

While you can create bare Pods with no problems, it is strongly recommended to make sure that the bare Pods do not have labels which match the selector of one of your ReplicaSets. The reason for this is because a ReplicaSet is not limited to owning Pods specified by its template-- it can acquire other Pods in the manner specified in the previous sections.

As those Pods do not have a Controller (or any object) as their owner reference and match the selector of the frontend ReplicaSet, they will immediately be acquired by it. [5]

 ownerReferences:
 - apiVersion: apps/v1
   blockOwnerDeletion: true
   controller: true
   kind: ReplicaSet
   name: frontend
   uid: f391f6db-bb9b-4c09-ae74-6a1f77f3d5cf

This information:

If you have multiple controllers that have overlapping selectors, the controllers will fight with each other and won't behave correctly. [6]

As we can see, it leads us to the warning:

Note: You should not create other Pods whose labels match this selector, either directly, by creating another Deployment, or by creating another controller such as a ReplicaSet or a ReplicationController. If you do so, the first Deployment thinks that it created these other Pods. Kubernetes does not stop you from doing this. [6]

As you can imagine you can create Replica Set with additional labels mentioned earlier f.e. pod-template-hash with the same value from another ReplicaSet managed by existing Deploymetnt and notice how it will crash.

The same is true if you apply this label to bare pods please see example in point 1.

In relation to Garbage Collection:

Garbage collection is a collective term for the various mechanisms Kubernetes uses to clean up cluster resources. [7]

Cascading deletion

Kubernetes checks for and deletes objects that no longer have owner references, like the pods left behind when you delete a ReplicaSet. When you delete an object, you can control whether Kubernetes deletes the object's dependents automatically, in a process called cascading deletion. There are two types of cascading deletion [8], as follows:

Foreground cascading deletion

Background cascading deletion

In background cascading deletion, the Kubernetes API server deletes the owner object immediately and the controller cleans up the dependent objects in the background. By default, Kubernetes uses background cascading deletion unless you manually use foreground deletion or choose to orphan the dependent objects. [7]

All references:

Why are my deployment/replicaset controllers not interfering with pods created with the same selectors?

1 Answers