Somehow identify celery workers that are shutting down gracefully

user_15704003 0
Viewed 309

I have three Celery workers as follows, each running on a different ECS node:

  • Producer: Keeps generating & sending tasks to the consumer worker. Each task is expected to take several minutes to compute and has a database record.
  • Consumer: Receives computation tasks and immediately starts execution.
  • Watchdog: Periodically inspects database records, finds out computation tasks that are executing, and then does celery inspect active to verify whether there is actually a worker carrying out the computation.

We ensured that when the Consumer node is being terminated, the Celery worker on it will begin graceful shutdown, so that the ongoing computation can finish normally. Because Celery will unregister a gracefully stopping worker, the consumer will become invisible to the Watchdog, who will mistakenly think a computation task has mysteriously lost... even though the Consumer is still working on the task.

Is it possible to let a Celery worker broadcast an "I am dying" message upon receiving a warm shutdown signal? Or even better, can we somehow let the Watchdog worker still see shutting workers?

celery python
1 Answers

Yes, it is possible. Nodes in Celery cluster I am responsible for are doing something similar. Here is a snippet:

@worker_shutdown.connect
def handle_worker_shutdown(**kwargs):
    _handle_worker_shutdown(app, _LOGGER, **kwargs)


@worker_ready.connect
def handle_worker_ready(**kwargs):
    _handle_worker_ready(app, _LOGGER, **kwargs)

There are few other, very useful signals that you should have a look, but these two are essential. Maybe the worker_shutting_down is more suitable for your use-case...

user_8764970
Related