Is it possible to set "worker_prefetch_multiplier" for each queue separately in Celery?

user_15202527 0
Viewed 978

I was wondering whether it is possible to set different prefetch multiplier for queues.

I have 2 queues, one has really short running tasks, other slightly longer. Queue for shorter tasks needs to be prioritized over other one.

To ensure that prioritization work reliable, this has to be set in celery config:

task_acks_late = True
worker_prefetch_multiplier = 1

However, that really hurts performance for fast task queue. Would it be possible to configure so that if worker is fetching from fast task queue, worker_prefetch_multiplier is 4 and if worker is fetching from slow task queuem worker_prefetch_multiplier is 1 ?

celery python-3.x python
1 Answers

I am not sure if it is possible to define different prefetch limits per queue since the Celery documentation seems to set these limits per worker.

However, we are solving this issue by starting a different worker for each queue. You can define different prefetch limits per worker - if one worker only uses one queue you can thus also define different prefetch limits as well as worker concurrencies per queue. This also has the added benefit that your long-running tasks would not block worker processing time for the short-running tasks. If you by any chance are thinking about using celery-batches to speed up processing for the short-running tasks even further, the queue separation into different workers becomes even more important since you want to then have quite high prefetch limits defined for that worker (note: you will eventually be running out of memory if your prefetch limit is 0 and you have a very full queue).

In our case, we are running our workers in a contianerized environment. This enables us to even define the resource allocation (memory / cpu) independent for each worker / queue.

user_13314070
Related