Can I set max task or max memory limits for gevent pools?

user_4379151 0
Viewed 956

We are using celery to run asynchronous tasks for a Django site.

The current worker setup is -pool=prefetch with --max-memory-per-child 5120000. The max memory threshold is important, because our tasks are leaking memory.

Now, we found in a recent analysis that our tasks are I/O bound and would work much better with a thread based execution pool like gevent, e.g. we can get much higher throughput.

However, neither the max-memory-per-child parameter nor the max-tasks-per-child setting are supported for thread based execution pools. The documentation says (source):

pool support: prefork

Is they any other celery configuration that could help me limit the max worker memory and/or force a worker restart after x task executions or is our only option to restart the worker with cron?

celery django python
1 Answers

While Celery does not support these settings for gevent, there is an alternative solution that I have been using in my production for some time now and which works well.

Supervisor with superlance

Assuming you are managing your celery workers through supervisor, there is a third party plugin called superlance, that adds a memory watchdog to supervisor.

The memory watchdog is called memmon. Memmon will watch the memory allocation of a program controlled by supervisor and automatically restart it if it surpassed a defined threshold.

Example

Here is an example supervisor configuration. worker is the celery worker program being monitored. Memmon is configured to check every 60 seconds and restart the worker if is surpasses 512MB. Note that Memmon in this configuration will do a "warm shutdown" of the worker, so no data will be lost.

[program:worker]
command=/home/allianceserver/venv/auth/bin/celery -A myauth worker -l info --pool=gevent --concurrency=10
directory=/home/allianceserver/myauth
user=allianceserver
numprocs=1
stdout_logfile=/home/allianceserver/myauth/log/worker.log
stderr_logfile=/home/allianceserver/myauth/log/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
killasgroup=true
priority=997

[eventlistener:memmon]
command=/home/allianceserver/venv/auth/bin/memmon -p worker=512MB
directory=/home/allianceserver/myauth
events=TICK_60
user_43791510
Related