I'm using the darknet repository from https://github.com/pjreddie/darknet
Using darknet, I can change my GPU at run-time where my model will run.
For example, the following code loads the darknet model weights on GPU 0, and I run prediction on 10 images.
from darknet import *
import time
# most likely this will be called when the server starts
set_gpu(0) # running on GPU 0, can change this
net1 = load_net(b"cfg/yolov3-lp_vehicles.cfg", b"backup/yolov3-lp_vehicles.backup", 0)
meta1 = load_meta(b"data/lp_vehicles.data")
# each time a request is sent
for i in range(10):
a = cv2.imread("lp_tester/bug1.jpg")
t1 = time.time()
r = detect_np_lp(net1, meta1, a) # detect_np_lp is just a custom function written to load the image as numpy array and pass to detector and get the predictions
t2 = time.time()
print(f"FPS: {1/(t2-t1)}")
darknet.py : https://github.com/pjreddie/darknet/blob/master/python/darknet.py
Now, I have 2 GPUs, I want to run two models in parallel. The one thing I realize for a starter is I need to load two models in two different GPUs when the program/server starts and keep them there as I can't load the model each time I send a request, loading is expensive.
How should I approach this? I'm not sure if multi-processing or multi-threading is the right option in this case (something that doesn't require any manual synchronization), here most of the computation will be done on GPU but I need two models running concurrently in two separate GPUs and after both of them are done, I need to merge the results in an array.
Any python code snippet demonstrating the similar behavior would be fine from which I can adapt, not necessarily have to be complete or consider the darknet scenario.