Which gunicorn worker type to use for machine learning inference?

user_1333610 0
Viewed 720
  • gunicorn version: 19.9.0
  • python version: 3.7.0

Got a named entity recognition (NER) machine learning inference application built with flask+gunicorn (no nginx).

The app receives a request with details of a document stored in the cloud. It fetches the document from the cloud, does NER on it, stores the results in the cloud (if it succeeds), and sends a success/failure response to the client. The SLA is the client receives a response within a minute of sending a request. The NER task runs multiple models in parallel using python multiprocessing and is quite CPU-intensive.

We are currently using a single gunicorn 'sync' worker with 7 threads. We are using only one worker, since the models take up a lot of memory. This set-up is mostly working alright, except that some threads just vanish in the middle of processing a request after fetching a document from the cloud (no errors in the logs), which requires us to restart gunicorn every few hours.

According to the gunicorn docs:

The default synchronous workers assume that your application is resource-bound in terms of CPU and network bandwidth. Generally this means that your application shouldn’t do anything that takes an undefined amount of time. An example of something that takes an undefined amount of time is a request to the internet.

Our app does make requests to the internet (cloud storage) and is also CPU-intensive.

What's the best worker class to use in this situation? Also, is there a better workers+threads combination?

gunicorn python-3.x
1 Answers

  1. When you use sync worker with 7 threads, Gunicorn automatically converts it to threads type worker. The presence of threads configuration will make Gunicorn switch the worker type

  2. You will likely see more improvements when using more workers. If your models are too big, consider using Tensorflow serving (https://www.tensorflow.org/tfx/serving/docker). Aside from other benefits like request batching, TF serving loads and runs the models in a separate process tensorflow_model_server, so only one copy of the ML models is needed, and that will free up your Gunicorn server so you can spring up more Gunicorn workers without worrying about each worker loads its own models into memory

  3. Now that you can use more workers, try with more sync workers with no threads configuration. sync worker is supposed to be good with CPU bound application.

  4. Then try with gevent workers no threads configuration. gevent worker uses pseudo-threads and is supposed to be very good with I/O bound application.

  5. Compare results with sync workers and gevent workers and see which one is better.

user_131016620
Related