Is it possible to use gevent execution pools with Celery beat?

user_13638086 0
Viewed 907

I have some tasks that I would like to run in a parallel fashion. They are almost 100% IO bound and not cpu intensive (simply making API requests).

I use celery/celerybeat in a Docker environment and have just added --pool=gevent to my docker-compose file, but I must also pass --beat.

When I start my app I get the following error:

celery.exceptions.ImproperlyConfigured: -B option doesn't work with eventlet/gevent pools: use standalone beat instead.

Is there any way to use thread execution with Celery beat?

celerybeat celery django python
1 Answers

What you are trying to do (i.e. embed beat inside the worker) isn't recommended anyway. So I'd suggest you start a standalone Celery beat (as adviced by the error message) and worker(s) with gevent pool. The worker's part is where you take advantage of gevent as they execute the I/O bound tasks, not the Celery beat that just kicks them off.

Alternatively, if you use Celery 4.4 or newer, you might try to use --pool threads to you threaded task pool.

user_58325400
Related