Celery --pool=threads -- what does this do and how do I use it properly?

Question

I'm hitting a segfault error while running a task using Celery. After looking up the issue, it seems others are solving similar issues by starting celery with --pool=threads.

When I try passing --pool=threads I get ModuleNotFoundError: No module named 'threads'

I don't believe this is the same as the thread module which would throw the error of No module named 'thread' instead.

How do I start using threads and what does it do?

The documentation regarding --pool=threads in the Celery site is very sparse. Searching for "--pool" will return nothing substantial, other than things related to billiard

user_3355367 · Answer

Just-released Celery 4.4.0 adds --pool thread to the concurrency list.

You can read more at Celery 4.4 Changelogs:

Threaded Tasks Pool

We reintroduced a threaded task pool using concurrent.futures.ThreadPoolExecutor.

The previous threaded task pool was experimental. In addition it was based on the threadpool package which is obsolete.

You can use the new threaded task pool by setting worker_pool to ‘threads` or by passing –pool threads to the celery worker command.

Now you can use threads instead of processes for pooling.

celery worker -A your_application --pool threads --loginfo=INFO

user_1741053 · Answer

If you want to use threads as your execution pool, you can choose between eventlet an gevent.

http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-p

You need to pip install eventlet/gevent to make it work.

Celery --pool=threads -- what does this do and how do I use it properly?

2 Answers