Serve Deep learning Model with Celery or RESTful API?

user_2291697 0
Viewed 1754

I am developing a web application which uses celery for task distribution and management. Web application also uses machine learning and deep learning algorithms to do predictions. These predictive models are deployed on separate server as separate application and their predictive functions are integrated with celery as individual task.

For example, X (user) wants to know forecast of stock price and submits query to web application. Web application will initiate celery tasks with X's query payload. This Celery tasks after performing certain operations submits tasks to other server where machine learning / deep learning 's celery is working and start waiting for other server tasks to be completed and response to be received.

This increased our performance to 10 folds as compare to when we deployed RESTful endpoints for machine learning predictive models using Flask. For deep learning, we need to move to Tensorflow and integrate it with celery. After thorough research, it was concluded to use Tensorflow Serving and call predictive functions inside celery tasks on machine learning server.

Other approach, was to deploy TensorFlow models as separate end points using Sanic and rather than web application's celery submitting tasks directly to other server celery, now it will directly execute and call RESTful API Endpoint which will be asynchronous as well.

What do you suggest, what would work best for us in this scenario? What benefit can celery provide over RESTful API or vice versa?

tensorflow deep-learning celery rest
1 Answers

For online processing of (mostly) serial requests within a cloud application, using Celery to queue up batches for inference doesn't seem like a good design choice, nor does the use of Flask, or any combination thereof. Tensorflow serving is a performant solution that handles dynamic batching for you, no need to put tasks on a queue that will then be sent to tf-serving only to again be queued prior to being processed.

Shameless plug: there is another package called virtex, built on top of asyncio, which offers a serving solution that is entirely agnostic to how your computation is implemented. It's very easy to use and highly performant. The server runs in a single process with an event loop handling requests. With this framework you define your request processing with three callback functions (which amounts to refactoring your inference code into three functions whose signatures are mildly constrained), and the virtex engine chains them together on its event loop. The virtex client can send requests in bundled or serial fashion. For online applications requiring low latency, in which the request/model-input ratio is roughly 1, it's the most performant http serving solution that I have tested (it's been benchmarked against tf-serving and bert-serving for Resnet50V2 and BERT-base, respectively).

user_61422690
Related