We’re using TorchServe on SageMaker with the deep learning containers ( https://github.com/aws/deep-learning-containers/blob/master/available_images.md).
Everything seems to be running well, we have very minimal CPU and memory usage, and our latency of requests can be about 3 seconds (which is quite long but okay). Once we start to ramp up usage our model latency increases dramatically - taking up to 1 minute to respond, however, our CPU and memory usage is still minimal... we feel like the TorchServe instance is not scaling properly on the instance and only deploying one worker when it should be at least 4 on a 4vcpu system (ml.m5xlarge instance). We have been following official AWS guides.
How can we resolve this?
We work with large files (1-20Mb images). However, SageMaker has a limitation of 5Mb payload size. To get around this we are using S3 to store our intermediary files (which potentially causes long latency). Our images go through preprocessing, enhancing, segmentation, and postprocessing, which are currently all individual SageMaker endpoints that a lambda function controls. We want to use SageMaker pipelines, but we see no incentive to make changes to our system with the payload limit.
Is there a better way of doing this?
We have been experimenting with Multi-Model instances. Ideally, we would like to have an ml.g4dnxlarge run all our PyTorch models to benefit from the low latency and high throughput. Still, when we deploy a torch serve multimode endpoint with deep learning containers, we get an error when invoking "model version not selected" ???? I have looked everywhere for anyone having similar issues but can't see anything.
Are there any guides for torch serve multi-model instances? Are we making an obvious mistake?
We have experimented with elastic inference and AWS Inf instances. However, we have seen no latency decrease from the standard CPU instance - again,
are we making a mistake here, and the benefits from these instances seem beneficial to us, and we would love to use them?