Python Multiprocessing Processes Go 100% to zero without finishing

user_3020085 0
Viewed 157

I am running Jupiter notebook to process some huge load on a list of arrays as:

import multiprocessing as mp
print("Number of processors: ", mp.cpu_count())

pool = mp.Pool(mp.cpu_count())
try:
    a_results = pool.map_async(process_each_area, [area for area in areas]).get()
except:
    print(e)
finally:
    pool.close()

Each processed area create one result file, which I expect to see once all the processes are done.

On submission, all 96 cpu core get busy and I see 96 new kernels as: enter image description here

$ ps -ef|grep ipykernel | wc -l 96

But after some time(80-90 minutes), all the cpu's go back to zero usage, while processes are still alive and not complete. I see then while grep as above. Also, results for all the areas are not created.

My question is: Why process go idle, but not complete after sometime? How can I debug them? Do we have threaddump in python?

jupyter-lab jupyter-notebook python-multiprocessing python-multithreading python
1 Answers

You have to actually return the result of the coroutine by calling get() on each one which runs the function. So instead of

try:
    a_results = pool.map_async(process_each_area, [area for area in areas]).get()
except:
    print(e)
finally:
    pool.close()

You need to do this:

try:
    coroutines = pool.map_async(process_each_area, areas)
    a_results = coroutines.get()
except:
    print(e)
finally:
    pool.close()

This way it will tell the AsyncResult object to return when the process is done. for more info refer to the docs

user_78852290
Related