Gist

When I start use copy-on-write to start up a bunch of wsgi workers, they freak out and use a ton of CPU and memory whenever I need to restart the main process. This causes OOM errors and I'd like to avoid it. I've tested in uwsgi and gunicorn and I see the same behavior.

Problem

I have a webapp with somewhat memory intensive process-based workers, so I use --preload in gunicorn and the default behavior in uwsgi so the application is fully loaded before forking in order to enable copy-on-write between the processes -- to save on memory usage.

When I shut down the main process (e.g. via a SIGINT or SIGTERM), all of the worker processes spike in CPU usage and my machine (Ubuntu 16.04, but also tested on Debian 10) loses a huge chunk of available memory. This often causes an OOM error. I don't see any rise in RES for each of the workers, but the drop in available mem roughly corresponds to what I would expect if all of the memory was being fully copied for each of the workers before being immediately de-allocated during their shut down. I'd like to avoid this sudden spike in memory usage and fully enjoy the benefits of copy-on-write.

Test environment

I have a really simple Flask app that you can use to test this:

from flask import Flask
application = Flask(__name__)
my_data = {"data{0}".format(i): "value{0}".format(i) for i in range(2000000)}
@application.route("/")
def index():
  return "I have {0} data items totalling {1} characters".format(
    len(my_data), sum(len(k) + len(v) for k, v in my_data.items()))

You can start the app with either of the following commands:

$ gunicorn --workers=16 --preload app:application
$ uwsgi --http :8080 --processes=16 --wsgi-file app.py

When I do ^C on the main process in my terminal and track the "free" KiB Mem reported by top, that's when I see the huge drop in available memory and the spike in CPU usage. Note that there is no change in memory usage reported for each worker. Is there a way to safely restart uwsgi/gunicorn so that this memory and CPU spike doesn't happen?

Steps to reproduce:

1. Set up app.py as described above
2. Run either gunicorn or uwsgi with the arguments provided above.
3. Observe free memory and CPU usage (using top)
   • 5.7GB free on my machine before startup
   • 5.3GB free on my machine after startup
4. Ctrl-C on the main gunicorn/uwsgi process
   • 1.3GB free while processes are shutting down (and CPU usage spikes)
   • 5.7GB free after all processes actually shut down (2-5 seconds later)