I have 3 separate Python processes, each running a TensorFlow session with my model loaded. The system has 1 GPU.
When I make a 3 predictions at the same time (1 per process), the prediction time is much slower than if I just make 1 prediction at a time.
I've limited the memory per model via the per_process_gpu_memory_fraction kwarg to 0.3.
Is there a way to get better performance (less time per response) when running models in parallel on a single GPU?