Celery is not executing tasks concurrently

user_9907206 0
Viewed 661

I'm new to celery, please let me know in comments if more information is required.

I have around 3000 tasks queued in redis and i want to execute these tasks concurrently over multiple threads, after a bit of research i ended up using eventlet for thread pooling and set concurrency to 500, like so

celery worker -A <app_name> -P eventlet -c 500

but when i checked celery flower many free threads are available which the celery is not using enter image description here

Any idea on how to make use of those free threads and make tasks faster? also if possible please suggest a good read for working with celery

celery redis django python
1 Answers

Each of those threads will run 500 eventlets. If your 3000 tasks are short-lived, they will be done with their work very fast. On top of that, your prefetch count is 400, which means while your worker (thread) runs 500 coroutines, it will also have 400 prefetched tasks. 3 * (500 + 400) = 2700 tasks will be taken from the queue immediately, assuming all your workers were idle.

user_8764970
Related