Failing on both Celery 4.4.7 and 5.0.2.
My task is leaking a substantial amount of memory.
I'd like to restart the worker after each task.
I'm starting the celery in a no concurrency mode --concurrency=1 or/and -P solo as the memory used in a single task might be close to the total system memory.
I tried to set the worker_max_tasks_per_child = 1 in the celeryconfig.py and --max-tasks-per-child=1 in CLI, but the worker is not restarting.
I am pretty sure that celery workers don't terminate after a task as ended. I would try using sys.exit at the end of your task, that way the process will exit and systemd or any other service in charge of your processes will bring it back up.
But the real solution should be to write a code with no memory leaks
In your case, the setting is not honored since you specified the pool configuration of your worker as 'solo'. It works only if your configuration is 'prefork'. If you start the worker with the changed configuration, you should be able to see the change. I used the option to prevent memory leak issue of tensorflow since it does not return the reserved memory till the process terminates, which was big problem in my application. I also didn't realize why the setting didn't work till see the limit from the link.
