Why is multiprocessing running and the CPU usage drops to 0%, but the results are never released?

Viewed 227

I have put together what I thought was some simple script to calculate the Moran's Index for each feature in a matrix (Information_Gains_Matrix), with its corresponding weight matrix (Weights_Matrix). I wanted to use pool.map to go along each feature in the Information_Gains_Matrix. The output is just a vector of Moran's Index's for each feature in Information_Gains_Matrix. See code:

import multiprocessing
from functools import partial 

def Feature_Moran_Index(Information_Gains_Matrix,Wij,N):
    Feature = Information_Gains_Matrix    
    X_bar = np.mean(Feature)
    if X_bar != 0:
        Deviance = Feature - X_bar
        Outer_Deviance = np.outer(Deviance,Deviance)
        Deviance2 = Deviance * Deviance
        Denom = np.sum(Deviance2)
        Moran_Index_Score = (N/Wij) * (np.sum((Weights_Matrix * Outer_Deviance))/Denom)
    else:
        Moran_Index_Score = 0
    return Moran_Index_Score

def Parallel_Feature_Moran_Index(Information_Gains_Matrix,Use_Cores):
    N = Information_Gains_Matrix.shape[0]
    Wij = np.sum(Weights_Matrix)  
    pool = multiprocessing.Pool(processes = Use_Cores)
    Result = pool.map(partial(Feature_Moran_Index, Wij=Wij,N=N), Information_Gains_Matrix)
    pool.close()
    pool.join()
    Moran_Index_Scores = np.asarray(Result)
    np.save("Moran_Index_Scores.npy",Moran_Index_Scores)
    return Result

if __name__ == '__main__':
    Moran_Index_Scores = Parallel_Feature_Moran_Index(Information_Gains_Matrix,Use_Cores=(multiprocessing.cpu_count()-2))

I'll openly admit my understanding of multiprocessing in Python isn't ideal, but I've used it a lot in the past with success. So I really don't understand why this simple script won't work. Help greatly appreciated. Note, I am aware the Weights_Matrix is not in the script. Its loaded in as a global so it doesn't have to be serialised to each core because it's very large. I'm sorry if this irritates anyone, I know some people get quite testy about the use of globals.

Edit: I've successfully used this exact code on smaller data sets on my local Ubuntu shell (Windows laptop). The problem occurs on a very large data set in a Ubuntu virtual machine.

0 Answers
Related