Celery: How to batch produce tasks?

user_9515210 0
Viewed 758

I have a large loop to produce tasks:

for i in range(1000):
    receiver.apply_async(args=(i), kwargs={}, exchange=topic_exchange, routing_key=topic_key)

And I found there is a module celery.contrib.batches before celery 3.X or celery_batches after celery 4.X. But this module doesn't seem to support such params. So how can I do it?

I'm using celery 4.4.7 with rabbitmq.

celery rabbitmq python
1 Answers

If by "batch" you mean a small subset (chunk) of the all tasks, then you could look into Chunks. Instead of using chunks (they are after all made for different purpose), I suggest you use Chord if you care about results. If you do not, then just simply create a Group. Thousand tasks is nothing - we have had chords/groups made of tens of thousands of tasks and Celery copes with that load pretty well.

user_8764970
Related