Multiple threads accessing same model on GPU for inference

user_10738278 0
Viewed 2453

I have a cnn model that is loaded onto the GPU and for every image, a new thread has to be created and detached to run the model on this image. Is this possible and if so, Is it safe?

pytorch python
1 Answers

Yes, you definitely can. There are two aspects to it. If you want to run each model in parallel, then you have to load the same model in multiple GPUs. If you don't need that (just want the threading part), then you can load the model and use concurrent.futures.ThreadPoolExecutor(). In each call, you can pass an image.

I demonstrated one example with the darknet framework.

I loaded the model in two separate GPUs (for parallel operation, you can avoid that too) and each time I get a request, I use ThreadPoolExecutor to pass the images to the processing function.

from darknet import *
import concurrent.futures
import time

# you can avoid this part if you don't need multiple GPUs
set_gpu(0) # running on GPU 0
net1 = load_net(b"cfg/yolov3-lp_vehicles.cfg", b"backup/yolov3-lp_vehicles.backup", 0)
meta1 = load_meta(b"data/lp_vehicles.data")

set_gpu(1) # running on GPU 1
net2 = load_net(b"cfg/yolov3-lp_vehicles.cfg", b"backup/yolov3-lp_vehicles.backup", 0)
meta2 = load_meta(b"data/lp_vehicles.data")


def f(x):
    if x[0] == 0: # gpu 0
        return detect_np_lp(net1, meta1, x[1])
    else:
        return detect_np_lp(net2, meta2, x[1])

    

def func2(): # with threading
    a1 = cv2.imread("lp_tester/bug1.jpg")
    a2 = cv2.imread("lp_tester/bug2.jpg")
    nums = [(0, a1), (1, a2)] # the first element in tuple denotes GPU ID
    with concurrent.futures.ThreadPoolExecutor() as executor:
        r_m = [val for val in executor.map(f, nums)]
    print('out f2')
    #return r_m

t1 = time.time()
func2()
t2 = time.time()
print(t2-t1)
user_46220460
Related