Sklearn VarianceThreshold not removing low variance features

user_14249902 0
Viewed 747

This is my first post here. If you have suggestions on asking questions more efficiently I'd like to hear it.

I am working with The Mercedez benz dataset, it can be found on kaggle here. This dataset has 369 numerical features. After removing the target variance and categorical features I am looking to remove the low variance features. I am using Sklearn's Variance Threshold.

I will include the code however these steps seem to be straight forward. I have played around with the threshold parameter but every time I pull the shape of the transformed dataset it has the same 369 features.

If anyone sees where I am going wrong I appreciate the help!

    X = df.iloc[:, df.columns != 'y']
    Y = df.iloc[:, df.columns == 'y']
    print(X.shape)
    print(Y.shape)

(4209, 377)
(4209, 1)

X_cat = X.select_dtypes(include = 'object')
X_num = X.select_dtypes(include = 'int64')
print(X_cat.shape)
print(X_num.shape)

(4209, 8)
(4209, 369)

X_num.var().sort_values()

X268    0.000000e+00
X297    0.000000e+00
X290    0.000000e+00
X289    0.000000e+00
X330    0.000000e+00
            ...     
X191    2.492121e-01
X362    2.496467e-01
X337    2.497867e-01
X127    2.500357e-01
ID      5.941936e+06
Length: 369, dtype: float64

from sklearn.feature_selection import VarianceThreshold
VT = VarianceThreshold()
VT.fit_transform(X_num)
print(X_num.shape)

(4209, 369)
feature-selection scikit-learn variance python
1 Answers

You did not transform your original data:

from sklearn.feature_selection import VarianceThreshold
# defining the function VT
VT = VarianceThreshold()
#Fit the function VT and transform, but not saving it
VT.fit_transform(X_num)

So you have to change it to:

from sklearn.feature_selection import VarianceThreshold
# defining the function VT
VT = VarianceThreshold()
#Fit the function VT and transform, and saving it in X_num
X_num = VT.fit_transform(X_num)
user_102329320
Related