I have put together what I thought was some simple script to calculate the Moran's Index for each feature in a matrix (Information_Gains_Matrix), with its corresponding weight matrix (Weights_Matrix). I wanted to use pool.map to go along each feature in the Information_Gains_Matrix. The output is just a vector of Moran's Index's for each feature in Information_Gains_Matrix. See code:
import multiprocessing
from functools import partial
def Feature_Moran_Index(Information_Gains_Matrix,Wij,N):
Feature = Information_Gains_Matrix
X_bar = np.mean(Feature)
if X_bar != 0:
Deviance = Feature - X_bar
Outer_Deviance = np.outer(Deviance,Deviance)
Deviance2 = Deviance * Deviance
Denom = np.sum(Deviance2)
Moran_Index_Score = (N/Wij) * (np.sum((Weights_Matrix * Outer_Deviance))/Denom)
else:
Moran_Index_Score = 0
return Moran_Index_Score
def Parallel_Feature_Moran_Index(Information_Gains_Matrix,Use_Cores):
N = Information_Gains_Matrix.shape[0]
Wij = np.sum(Weights_Matrix)
pool = multiprocessing.Pool(processes = Use_Cores)
Result = pool.map(partial(Feature_Moran_Index, Wij=Wij,N=N), Information_Gains_Matrix)
pool.close()
pool.join()
Moran_Index_Scores = np.asarray(Result)
np.save("Moran_Index_Scores.npy",Moran_Index_Scores)
return Result
if __name__ == '__main__':
Moran_Index_Scores = Parallel_Feature_Moran_Index(Information_Gains_Matrix,Use_Cores=(multiprocessing.cpu_count()-2))
I'll openly admit my understanding of multiprocessing in Python isn't ideal, but I've used it a lot in the past with success. So I really don't understand why this simple script won't work. Help greatly appreciated. Note, I am aware the Weights_Matrix is not in the script. Its loaded in as a global so it doesn't have to be serialised to each core because it's very large. I'm sorry if this irritates anyone, I know some people get quite testy about the use of globals.
Edit: I've successfully used this exact code on smaller data sets on my local Ubuntu shell (Windows laptop). The problem occurs on a very large data set in a Ubuntu virtual machine.