I am implementing a multiprocessing code for my python script. The way I am utilizing the multiprocessing module is as follows: -
with get_context("spawn").Pool() as p:
print('#########Starting Multiprocessing#########')
n_core = multiprocessing.cpu_count() if settings['ncore'] <= 0 else settings['ncore']
print('This is number of cores: ', n_core)
subteams = np.array_split(list(teams.values()), n_core)
func = partial(Team.bucketing, settings['bucket_size'], indexes['s2i'], indexes['c2i'], indexes['l2i'])
data = p.map(func, subteams)
I have tried running the job on different memory sizes - 85GB, 95GB, 120GB and 185GB. But everytime the job is getting stuck at one of the threads: -
Loading 29000/722593 instances by <SpawnProcess name='SpawnPoolWorker-4' parent=144941 started daemon>! 24144.20463013649
Loading 29500/722593 instances by <SpawnProcess name='SpawnPoolWorker-4' parent=144941 started daemon>! 24555.586552858353
Loading 30000/722593 instances by <SpawnProcess name='SpawnPoolWorker-4' parent=144941 started daemon>! 24960.114577770233
Loading 30500/722593 instances by <SpawnProcess name='SpawnPoolWorker-4' parent=144941 started daemon>! 25363.86053919792
Loading 31000/722593 instances by <SpawnProcess name='SpawnPoolWorker-4' parent=144941 started daemon>! 25775.94654607773
Loading 31500/722593 instances by <SpawnProcess name='SpawnPoolWorker-4' parent=144941 started daemon>! 26186.659839630127
Loading 32000/722593 instances by <SpawnProcess name='SpawnPoolWorker-4' parent=144941 started daemon>! 26599.891565561295
Loading 32500/722593 instances by <SpawnProcess name='SpawnPoolWorker-4' parent=144941 started daemon>! 27014.309255361557
Loading 33000/722593 instances by <SpawnProcess name='SpawnPoolWorker-4' parent=144941 started daemon>! 27420.721549749374
Could anyone please suggest how I should debug or figure out what is causing this issue? Should I just try with an even greater memory or is there any other way to resolve it? Any help would be greatly appreciated. Please let me know if there is anymore information that I can provide which might help you understand the issue. Thanks.