Passing function to map or starmap in multiprocessing pool

user_11295516 0
Viewed 515

i've been trying to test this simple piece of code on my machine on Jupyter and the cell just runs indefinitely without outputing anything. Is there some kind of bug or something? I use the exact same piece of code for a pandas process with pool.map and everything just works fine, but can't figure out what is happening here.

import multiprocessing as mp 
pool1 = mp.Pool(processes = 3)

def sumP(a, b):
    return (a * b) / (a - b + 1)

f1, f2, f3 = 24, 31, 45
new_rows2 = pool1.starmap(sumP, [(f1, f2), (f2, f3), (f1, f3)]) 
print(new_rows2)

And the cell just keeps running. Doesn't matter if i use pool1.map or pool1.starmap. Do you guys suggest any other way of parallelizing this process?

python-multiprocessing python-3.x multithreading python
1 Answers

This happens because the worker processes attempt to import sumP, which also creates another 3 processes, ad-infinitum.

You'll need to put a guard around the multiprocess creation so that workers don't spawn indefinitely:

import multiprocessing as mp 

def sumP(a, b):
    return (a * b) / (a - b + 1)

f1, f2, f3 = 24, 31, 45

if __name__ == "__main__":
    pool1 = mp.Pool(processes=3)
    new_rows2 = pool1.starmap(sumP, [(f1, f2), (f2, f3), (f1, f3)]) 
    print(new_rows2)

Look for "Safe importing of main module" under the multiprocessing docs for more info. You should not create a Pool during importing of the main module.

user_6740390
Related