Python Multiprocessing Erroneously Re-Running Entire File

user_14020477 0
Viewed 166

I'm experimenting with multiprocessing in python (on a windows pc) and wrote a test script to compare speeds of a simple procedure with and without multiprocessing. What I expected this to do was calculate three exponentials twice, once via multiprocessing and once normally, print "result" when each of the three multiprocessed calculations finished, and print the time taken for each of the two versions of the calculation. The intended output was:

result
result
result
multiprocessing took 0.2
normal processing took 0.4

What I actually get is:

multiprocessing took 0.0
multiprocessing took 0.0
normal processing took 0.013965368270874023
multiprocessing took 0.0
multiprocessing took 0.0
normal processing took 0.025931358337402344
multiprocessing took 0.0
multiprocessing took 0.0
normal processing took 0.009973287582397461
multiprocessing took 0.0
normal processing took 0.009966135025024414
multiprocessing took 0.0
normal processing took 0.00798344612121582
normal processing took 0.00798344612121582
normal processing took 0.00798177719116211
normal processing took 0.008982181549072266
result
result
result
multiprocessing took 0.5913138389587402
normal processing took 0.005980491638183594

It looks like the code is running repeatedly and in a weird order. Furthermore, if I include an else statement with the if __name__ == '__main__': which prints "oops" when __name__ is not '__main__', I see that the code is also running iterations where __name__ == '__main__' is not satisfied, which I didn't think was possible.

This is the code I was using, can anyone point out what is making it misbehave? Thanks in advance.

import concurrent.futures
import time

def function(x,y):
    return x**y

StartTime1 = time.time()

if __name__ == '__main__':
    with concurrent.futures.ProcessPoolExecutor() as executor:
        f1 = executor.submit(function, 3,4)
        f2 = executor.submit(function, 5,6)
        f3 = executor.submit(function, 7,8)

        for f in concurrent.futures.as_completed([f1,f2,f3]):
            print("result")

FinishTime1 = time.time()

print(f"multiprocessing took {FinishTime1-StartTime1}")

StartTime2 = time.time()

b1 = function(3,4)
b2 = function(5,6)
b3 = function(7,8)

FinishTime2 = time.time()

print(f"normal processing took {FinishTime2-StartTime2}")
python-multiprocessing python-3.x time multiprocessing python
2 Answers

When you import or load this file, all the code outside the if condition will be executed.

The proper way to structure the file is to place everything inside methods and then call these methods from the if condition

if __name__ == '__main__':
user_78058090

Hussein's solution has worked. For future askers, the adjusted code below has the correct output (note that all the outputs are now indented within the condition if __name__ == '__main__':)

import concurrent.futures
import time

def function(x,y):
    return x**y

if __name__ == '__main__':
    with concurrent.futures.ProcessPoolExecutor() as executor:
        StartTime1 = time.time()
        
        f1 = executor.submit(function, 3,4)
        f2 = executor.submit(function, 5,6)
        f3 = executor.submit(function, 7,8)

        FinishTime1 = time.time()

        for f in concurrent.futures.as_completed([f1,f2,f3]):
            print("result")


    

    print(f"multiprocessing took {FinishTime1-StartTime1}")

    StartTime2 = time.time()

    b1 = function(3,4)
    b2 = function(5,6)
    b3 = function(7,8)

    FinishTime2 = time.time()

    print(f"normal processing took {FinishTime2-StartTime2}")
user_140204770
Related