Error: selector does not match template labels

Viewed 48543

My task is to add a label named "app" to all deployments, daemonsets, and cronjobs so that it's easier to query our apps across the stack in our monitoring tools. This way, we can build dashboards that use a single selector, namely app.

To avoid downtime I've decided to resolve this issue in the following steps:

  1. Add labels to dev, test & stage environments.
  2. Add labels to prod env's.
  3. Deploy (1)
  4. Deploy (2)
  5. Delete old labels & update the services of dev to use the new labels. Then test & deploy. (currently on this step)
  6. Repeat (5) for stage.
  7. Repeat (5) for prod.

When using $ kubectl apply to update the resources I've added the "app" label to/replaced "service" label with "app" labels to, I run into the following error:

Error from server (Invalid): error when applying patch: {longAssPatchWhichIWon'tIncludeButYaGetThePoint} to: &{0xc421b02f00 0xc420803650 default provisioning manifests/prod/provisioning-deployment.yaml 0xc 42000c6f8 3942200 false} for: "manifests/prod/provisioning-deployment.yaml": Deployment.apps "provisioning" is invalid: s pec.template.metadata.labels: Invalid value: map[string]string{"app":"provisioning", "component" :"marketplace"}: selector does not match template labels

I need some insights on why it's throwing this error.

4 Answers

It seems you are in trouble. Check this section: Label selector updates

Note: In API version apps/v1, a Deployment’s label selector is immutable after it gets created.

So, this line say you can not update selector once deployment is created. Selector can not be changed for any API version except apps/v1beta1 and extension/v1beta1. Ref: TestDeploymentSelectorImmutability.

One possible workaround might be to keep the old labels and adding new labels along with old ones. This way, you don't have to update selector. Deployment will select pods using old labels but your dashboard can select using new labels. This might not meet your requirement but I don't see any better way.

This error is hard to read but it means that the labels specified in spec.template.metadata.labels of your Deployment definition do not match those of spec.selector.matchLabels within the same definition. Upload your YAML if you require further assistance. Best!

There are a few ways to resolve this from what I can tell. One way is to delete the deployment and re-apply the deployment with a key/value that works on your deployment:

spec:
  selector:
    matchLabels:
      app: app_name
  template:
    metadata:
      labels:
        app: app_name
        -- whatever else --

This obviously incurs downtime but should be permanent. Your other option is to edit the deployment selector:

kubectl -n namespace edit deployment app-deployment

Then run your apply command again. This may or may not be permanent as I don't know what changed the selector to begin with.

If your pod doesn't even exist yet to do modifications, this error might be legit. You may have a pod with the same name in the same namespace.

Someone came to me with this issue and it turned out that they had typed "matadata" instead of "metadata" so as far as kubernetes was concerned the label wasn't defined, which lead to this error message.

Related