Celery: How to allow queueing "duplicate" tasks, but disallow working on "duplicate tasks"?

Viewed 53

We're already using celery-singleton (similar to celery-once) to disallow queueing duplicate tasks, but now we have a case where we would like to allow queuing duplicate tasks but disallow working on duplicate tasks at the same time.

Example case: We have the job message {area_id: 42, item_id: 1} which is currently being worked on. For domain-specific reasons we would like to only have one task being worked on at the same time for any given area_id. The next message in the task queue is {area_id: 42, item_id: 2}. We would like this second task to wait until the first one is completed successfully. A task for a different area_id, e.g. {area_id: 23, item_id: 1} should be allowed to be executed at the same time by a different worker.

Are there any plugins to Celery similar to celery-singleton / celery-once that will easily allow us to accomplish this? I ask before trying to implement this myself in case a library already exists (which escaped my googling).

0 Answers
Related