I'm trying to figure out which is the best way to run multiple pre-trained models for inference in TensorFlow (in the context of single-machine execution). I read several questions about this, but I'm still a bit confused. For example, let us assume to run 2 deep networks in 2 different processes/threads. To my understanding, I can do this either:
- Running the two models within the same session (i.e., two processes which shares a single session), or
- running two different sessions, one for each network/process, or
- using TensorFlow Serving.
If I run this in a platform consisting of only a multi-core CPU, I imagine that the difference (from an execution point of view) is that with a single session there is a single intra op thread pool and a single inter pool thread pool, whereas in the second case they are distinct.
How does TensorFlow Serving modify the way in which the networks are executed on CPUs?
To my understanding, there is a benefit in using it when TensorFlow is used with GPUs, since it groups individual inference requests for joint execution.
In heterogenous architectures, does TF Serving adopt a different graph partitioning among device with respect to the one used in training (i.e., the one described in the TensorFlow White Paper)?
Among these three possibilities, which one is actually adopted in production environments?
If there are some specific documents that I missed which describe this characteristics of TensorFlow, please point me to all of them.