Celery task that runs more tasks

user_277256 0
Viewed 41294

I am using celerybeat to kick off a primary task that kicks of a number of secondary tasks. I have both tasks written already.

Is there a way to easily do this? Does Celery allow for tasks to be run from within tasks?

My example:

@task
def compute(users=None):
    if users is None:
        users = User.objects.all()

    tasks = []
    for user in users:
        tasks.append(compute_for_user.subtask((user.id,)))

    job = TaskSet(tasks)
    job.apply_async() # raises a IOError: Socket closed

@task
def compute_for_user(user_id):
    #do some stuff

compute gets called from celerybeat, but causes an IOError when it tries to run apply_async. Any ideas?

celery task django python
4 Answers

For the IOError mentioned, although the information here is not sufficient to tell what caused it, my guess is that you tried to establish a connection inside the task function, so whenever a task is called, a new connection is built. If the task is to be called thousand times, there will be thousand connections. This will flood the system socket manager and the IOError is its complaint.

user_87310950
Related