restricting tasks to run multiple times due to multiple pod restarts at same time

user_1611948 0
Viewed 56

I am new to Kubernetes and have been stuck at one point.

Lets say, I have multiple pods and I have some tasks running. Suddenly when pods stops due to whatever reason, I save the state of the task in some database (terminated but not completed) by catching SIGTERM signal or using terminationGracePeriod. So assuming I have 10 terminated tasks, I want to restart those tasks when the pod restarts. If multiple pods restarts, they all will fetch the terminated tasks from the database, makes the status "In Progress" and all will start the task. So instead of the task starting once it will start multiple times as multiple pods had found it terminated. I dont want to apply locks on database as it will slow down my code. So how can I restrict only one pod to fetch the terminated tasks and start only once.

Just FYI, I am trying to achieve the restarting of tasks in Golang.

kubernetes go
1 Answers

Store the state of the task in a database, and when the pod terminates, you update the state to 'terminated'. Then when pods start up again, query for tasks that have been 'terminated' and need to be continued. Grab a random ID for one of these tasks, and perform an UPDATE transaction to update the status to 'running' (make sure to also include WHERE status = 'terminated'). Single UPDATE operations in SQL are by default atomic, meaning no other transactions can modify the row while it is being updated. When using an ORM like GORM you will get a result containing the number of rows that was modified. If the number of rows is not equal to 1, that means another pod already updated this task, so we should grab another ID and try again until we perform an UPDATE where the number of rows updated is 1.

This is just an idea, no guarantees that this will work for you, as I do not know the full extent of your tech stack (what DB, ORM etc).

user_62527870
Related