I see that TorchServe can serve multiple models or multiple workers per model For example
this blog says "If your model is hosted on a CPU with many cores such as the c5.24xlarge EC2 instance with 96 vCPUs, you can easily scale the number of threads by using the method described previously"
So I see 3 things competing for cores:
- multiple models
- multiple workers per model
- multiple threads of the inference runtime running in each worker
How is TorchServe handling that triple level of parallelism? Is there any best practice or setting to tune?