Ignite baseline toplogy error joining nodes when restarted - Kubernetes

Question

I have the Ignite setup in Kubernetes. When the pods got restarted after some resource issue, the ignite node cannot join the cluster. Below is the error I got on one of the ignite pod

[ignite-0] Caused by: class org.apache.ignite.spi.IgniteSpiException: BaselineTopology of joining node (a5b9f9c1-c0ba-46c2-9608-ecda6e150183) is not compatible with BaselineTopology in the cluster. Branching history of cluster BlT ([424331775]) doesn't contain branching point hash of joining node BlT (-1658957222). Consider cleaning persistent storage of the node and adding it to the cluster again.

I am not sure what is the trouble here and could not fix this by deactivating & activating again, or enabling the auto adjust baseline. This is setup as statefulset, for your information.

How to solve this in case this happens on production?

user_11699511 · Answer

This means that the pod in question has a persistent store that is out of sync with the rest of the cluster.

Clean the persistent store on that pod:

replace the yaml image attribute to a linux image.
delete the appropriate persistent store dirs.
replace the image back to Ignite, and rescale the stateful set.

Use control.sh --baseline to determine the state of your baseline nodes.

Add/remove the appropriate node via control.sh commands if needed.

General information about baseline topology.

Ignite baseline toplogy error joining nodes when restarted - Kubernetes

1 Answers