"generic-garbage-collector" deleted StatefulSet and its pods. Why?

user_1168315 0
Viewed 214

The generic-garbage-collector deleted our StatefulSet prometheus-k8s and eventually also the corresponding pods. Which resulted in a ~5min outage of our monitoring system as all 3 pods were deleted at the same second.

Why did the generic-garbage-collector do this? What can it be?

The StatefulSet has this owner-reference which should stop deletion from garbage-collector (see K8s doc):

metadata:
  ownerReferences:
  - apiVersion: monitoring.coreos.com/v1
    blockOwnerDeletion: true
    controller: true
    kind: Prometheus
    name: k8s
    uid: 68631aa0-e528-11e9-8769-42010a8400bf

I.e. it is created & controlled by the prometheus-operator.

Kubernetes/GKE cluster version: 1.19

google-kubernetes-engine
1 Answers

The problem was triggered due to control plane downtime on 2021-05-24 17:03 PDT, which restarted the cluster master and forced the system to check each of the objects in the system to see if their owner is missing and rebuild that graph.

In case the problem happens again, the system will mitigate it automatically and it will provide more information to compare both incidents and try to find a root cause.

The best outcome is to migrate to 1.20.x where the problem is already fixed for sure.

user_157451050
Related