I am doing some NLP work and there are models trained by Tensorflow. I provide some APIs to accomplish word cut using these models and dispatching the requests through Celery.
The case is:
Celery will dispatch tasks to different workers(about 4-5) therefore each worker has to load the models above and consequently the Memory will be occupied a lot.
So any ways to share the models among the workers? I do not actually know about the underlying mechanism of Celery worker.
Thanks