Principal Component Analysis - Two approaches, yielding different results

Viewed 154

I'm trying to conduct Principal Component Analysis manually.

However, I've witnessed an issue:

Problem: when conducting PCA "manually" from scratch, using two different approaches, I don't get the same results

Let me showcase the issue by means of an example:

Consider the following numpy array (1):

x = np.array([
    [1.5, 2.3, 5.2, 3.2, 5.5],
    [3.5, 4.2, 6.5, 8.9, 7.5],
    [9.6, 8.2, 7.1, 9.3, 1.1],
    [3.1, 2.7, 2.9, 3.5, 9.6],
    [1.1, 6.7, 2.3, 3.5, 9.5]])

In the first approach, I first calculate the Covariance Matrix (2):

CovM = np.cov(x.T)

From the covariance matrix, I calculate the eigenvector and eigenvalues (3):

eig_vals, eig_vecs = np.linalg.eig(CovM)

I then sort my eigen values from highest to lowest (needed for PC calculation) (4):

eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]
eig_pairs.sort(key=lambda x: x[0], reverse=True)

I then store my eigenvectors (5):

matrix_w = np.hstack((eig_pairs[0][1].reshape(5,1), 
                  eig_pairs[1][1].reshape(5,1),
                  eig_pairs[2][1].reshape(5,1),
                  eig_pairs[3][1].reshape(5,1)
                ))

Calculating my principal components I get (6):

x.dot(matrix_w)

array([[ 1.57663265, -2.09793105,  6.2812378 ,  0.02538662],
   [ 5.18837375, -4.40212324, 11.50839994,  0.36124448],
   [13.58239293, -5.93222094,  6.90492473,  1.19626484],
   [-0.23287223, -5.19146231,  8.05750048,  3.01045736],
   [-0.33881008, -8.33915097,  7.44838336, -0.16441424]])

Now, note these results, they're going to be different from the second approach and that's exactly the issue! While they have the same magnitude they differ in signs (positive and negative values)

Now, for the second approach:

Consider the same numpy array from before (1). I will reuse that part, but calculate everything else from scratch, hence:

Covariance matrix for numpy array, x, can be found as: (7):

cov_mat = np.cov(x.T)

Similarily, we can calculate the eigenvalues and eigenvectors as before (8):

eigen_values, eigen_vectors = np.linalg.eigh(cov_mat)

Now, I can sort my eigenvalues from highest to lowest, using (9):

sorted_index = np.argsort(eigen_values)[::-1]
sorted_eigenvalue = eigen_values[sorted_index]
sorted_eigenvectors = eigen_vectors[:,sorted_index]

I choose to retain 4 components (first 4 PC), hence (10):

n_components = 4
eigenvector_subset = sorted_eigenvectors[:,0:n_components]

Similarly, as before, I can calculate the principal components (11):

np.dot(eigenvector_subset.transpose(),x.transpose()).transpose()

array([[ -1.57663265,   2.09793105,   6.2812378 ,   0.02538662],
   [ -5.18837375,   4.40212324,  11.50839994,   0.36124448],
   [-13.58239293,   5.93222094,   6.90492473,   1.19626484],
   [  0.23287223,   5.19146231,   8.05750048,   3.01045736],
   [  0.33881008,   8.33915097,   7.44838336,  -0.16441424]])

The above, shows the very issue. When comparing the results of (11) with (6), I see that I get different signs (positive and negative) for some reason that I can't explain - I tried to resolve the issue, one thing I noticed is if I use the definition of (3) i.e., eig_vals and eig_vecs instead of (8) when sorting in (9) I do indeed get the same results, but this seems a bit suspicious, given that the way I calculate the covariance matrices and the eigenvalues and eigenvectors are essentially the same in either approach, or am I missing something?

Please let me know if my question is not clear and consise

1 Answers

Solution: I found the issue, it relates to "eig" and "eigh" when calculating the eigenvector and eigenvalues. I was using either formulation in the two approaches, hence the different results.

Related