Problem: Multiprocessing in Python triggers other code simultaneously (concurrent.futures.ProcessPoolExecutor())

user_19740745 0
Viewed 25

I wrote this code to learn multiprocessing in Python.

import concurrent.futures
print("A")
tic = time.perf_counter()

def test():
    k = 1
    s = 0
    print(f"Starting ... ")
    for i in range(10000000):
        if i % 2 == 0:
            s += 4/k
        else:
            s -= 4/k
        k += 2
    print(f"... finished!")
    
def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.submit(test)
        executor.submit(test)

if __name__ == '__main__':
    main()

toc = time.perf_counter()
print(f"Took {toc - tic:0.4f} seconds")

I would expect a plain output of the seconds but it instead runs print("A") three times too:

A
A
Took 0.0000 seconds
Starting ...
A
Took 0.0000 seconds
Starting ...
... finished!
... finished!
Took 4.0650 seconds

I looked up various examples across google but can't seem to nail down the problem. Any suggestions?

multiprocessing python
1 Answers

Because you put your prints of A and the timing outside of the main function, these commands are run once, when you start the program and for every process you start of executor.submit(test). This is intended as the ProcessPoolExecutor creates new instances that import itself into the new process and therefore executes all global variables for every instance.

It is also in the documentation:

The main module must be importable by worker subprocesses. This means that ProcessPoolExecutor will not work in the interactive interpreter.

Documentation: https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor

To solve it, move your print statements inside the main function like this:

import concurrent.futures, time

def test():
    k = 1
    s = 0
    print(f"Starting ... ")
    for i in range(10000000):
        if i % 2 == 0:
            s += 4/k
        else:
            s -= 4/k
        k += 2
    print(f"... finished!")
    
def main():
    pass
    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.submit(test)
        executor.submit(test)

if __name__ == '__main__':
    print("A")
    tic = time.perf_counter()
    main()
    toc = time.perf_counter()
    print(f"Took {toc - tic:0.4f} seconds")

Result:

A
Starting ... 
Starting ... 
... finished!
... finished!
Took 3.7274 seconds

You can also switch to ThreadPoolExecutor instead, which does not import itself.

user_113559260
Related