I'm wondering what my best bet is for deploying spacy in the cloud for real-time inference, with a serverless solution?
Basically, I want an endpoint similar to what AWS can offer with an EC2 or SageMaker instance, but at the smallest possible cost and with autoscaling (to zero in times of no traffic).
I can tolerate a few seconds delay for cold starts (when loading the model), but after that inference should be at basically no overhead for model loading. Hence AWS Lambda is not an option as it won't maintain the model in memory between invocations of the function (or so I believe?). I have looked into SageMaker Serverless Inference, but find it quite poorly documented, not sure if applicable for my use case?