Multiprocessing only running the function and not the entire file

user_11945801 0
Viewed 661

If I run the following code, it would run the print("a") for every time that the pool.map runs.

from multiprocessing import Pool
from os import getpid

print("a")

def double(i):
    print("I'm process", getpid())
    return i * 2

if __name__ == '__main__':
    with Pool() as pool:
        result = pool.map(double, [1, 2, 3, 4, 5])
        print(result)

Normally this wouldn't be an issue, but in the code that I want to implement multiprocessing in has a long start up time which makes multiprocessing take a long time to get the result, which cancels out the time save of using multiprocessing. So is there a way to make it run only the function in parallel and not the entire file?

multiprocessing python
2 Answers

You can use multiprocessing.pool.ThreadPool to run the function inside one process:

from multiprocessing.pool import ThreadPool
from os import getpid

print("a")

def double(i):
    print("I'm process", getpid())
    return i * 2

if __name__ == '__main__':
    with ThreadPool() as pool:
        result = pool.map(double, [1, 2, 3, 4, 5])
        print(result)

Prints (note the "garbled" output):

a
I'm processI'm process  I'm process 1
I'm process 1
I'm process1
 1
1
[2, 4, 6, 8, 10]
user_100359850

this is one of the major drawbacks of multiprocessing. The problem with starting an entire new python process is that it requires that overhead to re-import every module. If you are simply looking for a way to parallelize tasks, look into using concurrent.futures.

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=5) as executor:
    futures = executor.map(double, [1, 2, 3, 4, 5])
    print([f for f in futures])

You could also try to use multiprocessing on a larger portion of calculations!

user_140557520
Related