Multiprocessing: pool and map and sys.exit()

user_1251660 0
Viewed 2932

I think I need some suggestion here. Below is my code:

from multiprocessing import Pool
import time
import sys

def testing(number):
    count = 0
    while True:
        print('Count: {}'.format(count))
        count += 1

        if count > number:
            print('Exiting...')
            sys.exit()
        else:
            print('Looping Over')
            time.sleep(1)

if __name__ == '__main__':

    with Pool(2) as p:
        p.map(testing, [3, 2])

EXPECTED OUTCOME:

The program (main thread) should exit once all child threads have exited.

ACTUAL RESULT:

$ python3 test_exit.py
Count: 0
Looping Over
Count: 0
Looping Over
Count: 1
Looping Over
Count: 1
Looping Over
Count: 2
Looping Over
Count: 2
Exiting...   <<< Exited 1st thread.
Count: 3
Exiting...   <<< Exited 2nd thread.
....and it stays here as if stuck or something. It never gives control back to Shell.

EXPECTED RESULT:

$ python3 test_exit.py
Count: 0
Looping Over
Count: 0
Looping Over
Count: 1
Looping Over
Count: 1
Looping Over
Count: 2
Looping Over
Count: 2
Exiting...
Count: 3
Exiting...
$   <<< Note: I am expecting to be dropped back to Shell prompt

QUESTION:

Is there something wrong in my approach in term of pool/map usage?

multiprocessing python
3 Answers

The program (main thread) should exit once all child threads have exited.

  • finish a process with terminating its target function testing() (done through break statement in the crucial loop)
  • exit main thread/program when the pool of processes has been completed.
from multiprocessing import Pool, current_process
import time
import sys

def testing(number):
    count = 0
    while True:
        print('Count: {}'.format(count))
        count += 1

        if count > number:
            print('Exiting...', current_process().name)
            break
        else:
            print('Looping Over')
            time.sleep(1)

if __name__ == '__main__':

    with Pool(2) as p:
        p.map(testing, [3, 2])
    sys.exit()

The output:

Count: 0
Looping Over
Count: 0
Looping Over
Count: 1
Looping Over
Count: 1
Looping Over
Count: 2
Looping Over
Count: 2
Exiting... ForkPoolWorker-2
Count: 3
Exiting... ForkPoolWorker-1
$
user_31854590

an explanation of this behaviour:

This happens because when you call sys.exit() it raises systemExit Exception.since sys.exit() ultimately only raises an exception, it will only exit the process in which is called and is not propagated further up to main processes.

Once all child processes exit the main process just sits there waiting for something to be returned from the child process. All child processes have already exited so there is nothing to return resulting in a forever wait

user_65181180

Try os._exit(1) instead of sys.exit();

user_92487480
Related