I am new to Celery and I am trying to understand how queues work. If I have two tasks say task1 and task2 and I place them in different queues and on task1 I use only a single worker and on task2 I use multiple works, will task1 only run one at a time since I only have a single worker? and will task2 run as many as many times as I have workers? Is my understanding correct?
2 Answers
You are close. You can distribute tasks to specific queues and configure workers to only listen to specific queues and scale the number of workers listening to each queue independently. More workers, generally, means more tasks can execute concurrently.
However, having only a single worker assigned to a particular queue/task alone does not guarantee that the task will only execute one at a time.
By default, workers have concurrency enabled, meaning a single worker can utilize multiple processes to execute tasks concurrently. Further, there are other working settings to consider, too, like prefetching and early acknowledgement.
If you want to ensure that a task can only be executed one at a time, you should not rely on the (lack of) availability of worker processes. Instead, a locking mechanism like described in the docs ensuring a task is executed one at a time would be one recommended approach for this.
task2 will not be executed as many times as you have workers on that queue, this task will be distributed to one of workers that will be available at that moment of time.
Related