Tensorflow Serving Fails if a new request comes before previous computation is finished

user_6364229 0
Viewed 744

I currently have a web api run using flask, gunicorn, and nginx, the web api calls my tensorflow model serving port. When the web api is called multiple times before the first is finished the tensorflow model fails gives an empty request.

What is the best way to handle this? My web api is currently already behind gunicorn and nginx, but the calls from the api to the tensorflow model seems to be the problem. Should I put it behind gunicorn / nginx load balancer as well?

Thanks

tensorflow-serving gunicorn load-balancing python
1 Answers

You have several options.

  1. Use several workers. Each gunicorn worker will load a model. This must be done on start, because it takes some time to load a model. Workers can handle several petitions.
  2. Use a state machine: idle, processing. You will raise exceptions if a you receive a GET when processing and raise 503 Service Unavailable.
user_7448590
Related