How to RAM efficiently load spacy models into fastapi with gunicorn?

user_4755832 0
Viewed 751

I am having challenges with my fastapi server running out of RAM after updating to use the spacy_en_core_web_lg from the small model.

When running fastapi 4 gunicorn workers are spawned and based on the memory usage I think each worker is loading in the model. Is there a way I can share the model across workers so I don't need to load it in each?

gunicorn fastapi spacy
1 Answers

The following is what has helped in my case. YMMV, particularly because I have not used Spacy directly but PyTorch.

I wrote a longer form post about this topic here: http://www.streppone.it/cosimo/blog/2021/08/deploying-large-deep-learning-models-in-production/

Here's a summary:

  • Use gunicorn preload_app = True option, to have gunicorn load your application before the workers fork()
  • Load the model before the FastAPI application is created
  • If the model is PyTorch based, use model.eval() and model.share_memory(). See further documentation here: https://pytorch.org/docs/stable/multiprocessing.html
  • Limit the amount of workers (which you are already doing). Three workers was best for me. Four also seems reasonable, it really depends on your project and requirements
  • Limit the lifetime of each worker through gunicorn max_requests. In my case, I noticed a sharp increase of memory usage for each worker after some time, so this option curbs that behaviour.

Further links and reading in the post I mentioned. I would be very interested in feedback about these tips, because so far I haven't been able to find any good references online.

user_113030
Related