i am using celery for some tasks that no need to much memory (something like 2 MB is enough . more of task is loop with sleep 10 sec) but i have 100 or more tasks like this and this tasks giving so many times ( some times 10 days ) and data not stored in memory because it will saved in data base so no need many memory but every worker of celery giving so huge of server memory ( like 100 MB per worker )
so i search and see some this
it suggested to use gevent or evenlet but this pools not supporting killing(revoke()) task ( i need that )
and other solution was using auto-scale but it don`t make memory usage lower ( at 100 task no help )
so i decide to set max memory usage and test bellow code but not work
in celery_config.py
app = Celery('myproj')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.conf.update(worker_max_memory_per_child=8000) # 8MB
so i try other code as bellow and not work again
in settings.py
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Asia/Tehran'
# above codes working but bellow not
CELERY_TASK_TRACK_STARTED = True
so i try other code
in supervisord.d/myproj_celery.conf
command=/var/www/myproj/venv/bin/celery worker -A myproj --loglevel=INFO --max-memory-per-child=2000 --autoscale=15,2 --concurrency=5
# autoscale work but memory usage of every worker still 100MB
any solution or any lead to new way??
i am using python3.8 and celery 4.4.0 and django==2.1.* and server centos 7