I am trying to create a table of pairwise correlation for a model that I am building, and I have some numpy.nan values (NAN) in my dataset. For some reason, when I perform the correlation using np.corrcoef() I have different results than using pd.df.corr():
for instance:
dataset = np.array([[1,np.nan,np.nan,1,1],[1,np.nan,np.nan,3000,1]])
pandas_data = pd.DataFrame(dataset.transpose())
print np.corrcoef(dataset)
to which I get:
[[ nan nan]
[ nan nan]]
but with the pandas dataframe I do have one result:
print pandas_data.corr()
0 1
0 NaN NaN
1 NaN 1
Is there a fundamental difference in the way they handle NaN, or I missed something? (Also, why is my correlation 1 if I do have different values?) Thanks