Sklearn TruncatedSVD not showing explained variance ration in descending order, or first number means something else?

user_8331923 0
Viewed 169
from sklearn import datasets
from sklearn.decomposition import PCA
from sklearn.decomposition import TruncatedSVD 

digits = datasets.load_digits()
X = digits.data
X = X - X.mean() # centering the data

#### svd
svd = TruncatedSVD(n_components=5)
svd.fit(X)
print(svd.explained_variance_ration)

####  PCA
pca = PCA(n_components=5)
pca.fit(X)
print(pca.explained_variance_ratio_)

svd output is:

array([0.02049911, 0.1489056 , 0.13534811, 0.11738598, 0.08382797])

pca output is:

 array([0.14890594, 0.13618771, 0.11794594, 0.08409979, 0.05782415])

is there a bug in the TruncatedSVD implementation? or why is the first explained variance (0.02...) behaving like this? or what is the meaning

svd scikit-learn machine-learning python-3.x
1 Answers

Summary:

That is because TruncatedSVD and PCA use different SVD functions!.

Note: Your case is due to Reason 2 below, yet I included another reason for future readers.

Details:

Reason 1: The solver set by user in each algorithm, is different:

PCA internally uses scipy.linalg.svd which sorts singular values, hence the explained_variance_ratio_ is sorted.

Part of Scikit Implementation of PCA:

# Center data
U, S, Vt = linalg.svd(X, full_matrices=False)
# flip eigenvectors' sign to enforce deterministic output
U, Vt = svd_flip(U, Vt)

components_ = Vt

# Get variance explained by singular values
explained_variance_ = (S ** 2) / (n_samples - 1)
total_var = explained_variance_.sum()
explained_variance_ratio_ = explained_variance_ / total_var

Screenshot from the above-mentioned scipy.linalg.svd link:

enter image description here

On the other hand, TruncatedSVD uses scipy.sparse.linalg.svds which relies on the ARPACK solver for decomposition.

Screenshot from the above-mentioned scipy.sparse.linalg.svds link:

enter image description here

Reason 2: The TruncatedSVD operates differently compared to PCA:

In your case you chose randomized as a solver (which is set by default) in both algorithms, yet you obtained different results with regards to the order of the variance.

That is because in PCA, the variance is obtained from the actual singular values (called Sigma or S in Scikit-Learn implementation), which are already sorted:

enter image description here

On the other hand, the variance in TruncatedSVD is obtained from X_transformed which results from multiplying the data matrix by the components. The latter does not necessarily preserve order because data are not centered, nor is it the purpose of TruncatedSVD which it is used in first place for sparse matrices:

enter image description here

Now if you center your data, you will get them sorted (note that you did not center data properly, because centering requires dividing by standard deviation):

from sklearn import datasets
from sklearn.decomposition import TruncatedSVD
from sklearn.preprocessing import StandardScaler

digits = datasets.load_digits()
X = digits.data

sc = StandardScaler()
X = sc.fit_transform(X)

### SVD
svd = TruncatedSVD(n_components=5, algorithm='randomized', random_state=2021)
svd.fit(X)
print(svd.explained_variance_ratio_)

Output

[0.12033916 0.09561054 0.08444415 0.06498406 0.04860093]

Important: Further read.

user_63651120
Related