How to prevent repeated reconcile calls

user_1632732 0
Viewed 562

I'm developing a Kubernetes controller. The desired state for this controller is captured in CRD-A and then it creates a deployment and statefulset to achieve the actual state. Currently I'm using server side apply to create/update these deployment and statefulsets.

The controller establishes watch on both CRD-A as well as deployments, statefulset. This to ensure that if there is a change in the deployment/statefulset, the reconcile() is notified and takes action to fix it. Currently the reconcile() always calls server side apply to create/update and this leads another watch event (resource version changes on every server side apply) resulting in repeated/infinite calls to reconcile()

One approach I've been thinking about is to leverage 'generation' on deployment/statefulset i.e. the controller will maintain a in-memory map of (k8s object -> generation) and on reconcile() compare the value in this map to what is present in the indexed informer cache; do you see any concerns with this approach? And are there better alternatives to prevent repeated/infinite reconcile() calls?

kubernetes
2 Answers

One idea could be to not apply the objects if there are no changes to be applied. This gives you the problem of how to detect if the objects are up to date already, but it's hard to tell if that's doable in your case.

user_10789690

Ideally, if the object you provide in the server side apply is not changed, the generation and the resourceVersion of the object should BOTH NOT be changed.

But sometimes that's not the case, see this github issue:https://github.com/kubernetes/kubernetes/issues/95460

Fortunately, the generation always stays the same, so yes, you can leverage this field to avoid the reconcile dead loop, by adding a GenerationChangedPredicate filter to your controller, which will skip reconciling if the generation does not change, and it's often used in conjuction with the LabelChangedPredicate, which filters events when the object's labels does not change.

Here's how you will set up your controller with these two predicates:

ctrl.NewControllerManagedBy(mgr).
    For(&Object{}).
    Owns(&appsv1.StatefulSet{}).
    Owns(&appsv1.Deployment{}).
    // together with Server Side Apply
    // this predicates prevents meaningless reconcilations from being triggered
    WithEventFilter(predicate.Or(predicate.GenerationChangedPredicate{}, predicate.LabelChangedPredicate{})).
    Complete(r)
user_163800350
Related