When to Fan Out with Celery Groups vs. Delay

user_7113209 0
Viewed 403

Trying to figure out best practice here as I am learning celery.

I want to fan out a bunch of small jobs in another job. I understand groups is a great way to get the results from a bunch of enqueued smaller jobs, but what if I don't care about results? I just want to split a big job into a bunch of smaller tasks. Is calling delay in a loop the same thing in this case?

Is it still best practice to use groups when wanting to fan out, and queue up a bunch of smaller jobs in a parent job? Below is some pseudo code as an example.

@app.task 
def call_job(job_args):
    groups([for small_job.s(x) for x in job_args]).delay()

@app.task
def small_job(x):
    # do something

vs 

@app.task 
def call_job(job_args):
    for x in job_args:
        small_job.delay(x)

@app.task
def small_job(x):
    # do something
celery django python
1 Answers

I think you should go with the second approach (for loop):

  • Use group when you want to know that all smaller tasks finished (because you want to see their result together).
  • Use group when another action/task should happen only after all tasks finished (chord).

Behind the scene, someone (Celery backend) needs to take care of managing the finished tasks so it can answer whether the group results are ready().

The bottom line, if you don't care about your tasks' return value and there is no dependency between finished them all and a follow-up action - go with the for loop.

user_10112530
Related