In my program I need to spawn 16 multiprocessing pools, to utilise 16 cores on my 3950x. I have an initialiser which initialises three global variables in the spawn child processes (totalling around 300kb).
Before the using the initialiser each process took about 1s to spawn. Now 16 process takes around 100s in total! Any idea why this is now so slow? Sample code below:
def set_mp_global_vars(bc, bm, bf_n):
"""mp initialiser which sets global vars to reduce mp overhead"""
global base_comps, base_matches, bf_names
base_comps = bc
base_matches = bm
bf_names = bf_n
int_pool_workers = mp.cpu_count()
pool = mp.Pool(processes=int_pool_workers, initializer=set_mp_global_vars,
initargs=(base_comps, base_matches, bf_names))