Why does the K3s system upgrade controller fail with "not found, requeuing"?

Viewed 26

I installed the system upgrade controller and applied this plan manifest:

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: master-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: k3s-master-upgrade
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/stable
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: worker-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: k3s-worker-upgrade
      operator: In
      values:
      - "true"
  prepare:
    args:
    - prepare
    - master-plan
    image: rancher/k3s-upgrade
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/stable

I applied and checked the labels:

$ kubectl label node crux k3s-worker-upgrade=true
$ kubectl describe nodes crux | grep k3s-worker-upgrade
                    k3s-worker-upgrade=true

$ kubectl label node nemo k3s-master-upgrade=true
$ kubectl describe nodes nemo | grep k3s-master-upgrade
                    k3s-master-upgrade=true

According to kubectl get nodes I'm still on v1.23.6+k3s1, but the stable channel is on v1.24.4+k3s1.

I get the following errors:

$ kubectl -n system-upgrade logs deployment.apps/system-upgrade-controller

time="2022-09-12T11:29:31Z" level=error msg="error syncing 'system-upgrade/apply-worker-plan-on-crux-with-4190e4adda3866e909fc7735c1-f0dff': handler system-upgrade-controller: jobs.batch \"apply-worker-plan-on-crux-with-4190e4adda3866e909fc7735c1-f0dff\" not found, requeuing"
time="2022-09-12T11:30:35Z" level=error msg="error syncing 'system-upgrade/apply-master-plan-on-nemo-with-4190e4adda3866e909fc7735c1-9cf4f': handler system-upgrade-controller: jobs.batch \"apply-master-plan-on-nemo-with-4190e4adda3866e909fc7735c1-9cf4f\" not found, requeuing"
$ kubectl -n system-upgrade get jobs -o yaml

- apiVersion: batch/v1
  kind: Job
  metadata:
    labels:
      upgrade.cattle.io/controller: system-upgrade-controller
      upgrade.cattle.io/node: crux
      upgrade.cattle.io/plan: worker-plan
      upgrade.cattle.io/version: v1.24.4-k3s1
  status:
    conditions:
    - lastProbeTime: "2022-09-12T12:14:31Z"
      lastTransitionTime: "2022-09-12T12:14:31Z"
      message: Job was active longer than specified deadline
      reason: DeadlineExceeded
      status: "True"
      type: Failed
    failed: 1
    startTime: "2022-09-12T11:59:31Z"
    uncountedTerminatedPods: {}
0 Answers
Related