Why is Pandas and Numpy producing different results for pairwise correlations with NaN?

Viewed 1472

I am trying to create a table of pairwise correlation for a model that I am building, and I have some numpy.nan values (NAN) in my dataset. For some reason, when I perform the correlation using np.corrcoef() I have different results than using pd.df.corr():

for instance:

dataset = np.array([[1,np.nan,np.nan,1,1],[1,np.nan,np.nan,3000,1]])
pandas_data = pd.DataFrame(dataset.transpose())

print np.corrcoef(dataset)

to which I get:

[[ nan  nan]
[ nan  nan]]

but with the pandas dataframe I do have one result:

print pandas_data.corr()

    0   1
0 NaN NaN
1 NaN   1

Is there a fundamental difference in the way they handle NaN, or I missed something? (Also, why is my correlation 1 if I do have different values?) Thanks

1 Answers
Related