How to monitor a group of tasks in celery?

user_4709400 0
Viewed 1884

I have a situation where a periodic monthly big_task reads a file and enqueue one chained-task per row in this file, where the chained tasks are small_task_1 and small_task_2:

class BigTask(PeriodicTask):

    run_every = crontab(hour=00, minute=00, day_of_month=1)

    def run(self):
        task_list = []
        with open("the_file.csv" as f:
            for row in f:
                t = chain(
                        small_task_1.s(row),
                        small_task_2.s(),
                     )
                task_list.append(t)
            gr = group(*task_list)
            r = gr.apply_async()

I would like to get statistics about the number of enqueued, failed tasks (and detail about the exception) for each small_task, as soon as all of them are finished (whatever the status is) to send a summary email to the project admins.

I first thought of using chord, but callback is not executed if any of the headers task fails, which will surely happen in my case.

I could also use r.get() in the BigTask, very convenient, but not recommended to wait for a task result into another task (even if here, I guess the risk of worker deadlock is poor since task will be executed only once a month).

Important note: input file contains ~700k rows.

How would you recommend to proceed?

celery python
3 Answers

I found the best solution to be iterate over the group results, after the group is ready.

When you issue a Group, you have a ResultSet object. You can .save() this object, to get it later and check if .is_ready, or you can call .join() and wait for the results.

When it ends, you can access .results and you have a list of AsyncResult objects. These objects all have a .state property that you can access and check if the task was successul or not.

However, you can only check the results after the group ends. During the process, you can get the value of .completed_count() and have an idea of group progress.

https://docs.celeryproject.org/en/latest/reference/celery.result.html#celery.result.ResultSet

user_16931890
Related