Celery - Check if worker received SIGTERM

user_972014 0
Viewed 2435

I have a Celery task that is quite long. More than a few minutes.

Sometimes, for various reasons, a worker is marked for termination and another worker starts. This can happen if the machine it runs on needs to be replaced, or a new code version is being deployed. In that case, the worker receives the SIGTERM signal.

I'm wondering if it's possible for the task itself to periodically check whether this worker has received SIGTERM and is pending termination, and in that case just put the task back in the queue and terminate. (The task will then be started on another worker, and will continue doing its work)

EDIT: Clarifying - Is it possible within the task to check whether it is executed on a worker that waits to be terminated. Like this:

# Some long task that can take even a few hours.
def some_task(...):
    for i in range(...):
        do_some_work()
        # That's the missing function:
        if did_this_worker_received_SIGTERM_and_waiting_to_be_terminated():
             # stop the task in the middle, and it will be executed again later
celery-task celery python
2 Answers

When Celery worker receives SIGTERM, it will initiate the warm shutdown. It means it will unsubscribe itself from all the queues, prefetched tasks (if any) will go back to their queues, and worker itself will start waiting for the currently running tasks to finish before it shuts down. No task is lost, if that is what you fear.

All these events can be handled (see Worker Signals).

If you still insist on having some extra logic inside your tasks that handles worker state, then perhaps the easiest solution is to implement worker shutdown handler (as described in the doc section I mentioned above), make it store a flag in Redis or some other distributed K/V storage), and refactor your tasks that need this so they access this flag and perform whatever action you need them to perform.

user_8764970

Can I ask why do you want to do such a thing? did you enable task_acks_late? That way, if a task won't finish on time and the worker will go down, the task will rerun again on a new worker.

Here is the documentation. There's also task_reject_on_worker_lost which I didn't try, but maybe it can help you as well:

Setting this to true allows the message to be re-queued instead, so that the task will execute again by the same worker, or another worker.

user_10112530
Related