How does TorchServe prevent CPU contention when using multi-model or multi-workers?

Viewed 15

I see that TorchServe can serve multiple models or multiple workers per model For example

this blog says "If your model is hosted on a CPU with many cores such as the c5.24xlarge EC2 instance with 96 vCPUs, you can easily scale the number of threads by using the method described previously"

So I see 3 things competing for cores:

  1. multiple models
  2. multiple workers per model
  3. multiple threads of the inference runtime running in each worker

How is TorchServe handling that triple level of parallelism? Is there any best practice or setting to tune?

0 Answers
Related