How to properly do feature selection with SelectFromModel from scikit-learn?

user_7762646 0
Viewed 4648

I am using a very simple kaggle dataset to understand how SelectFromModel with a logistic regression works. The idea was to create a very simple pipeline with some basic data processing (dropping a column + scaling), pass it to feature selection (logreg) and then fit an xgboost model (not included in the code). By reading the documentation, my understanding is that given my X_train and y_train a logreg model is fitted and those features with coeffs higher or equal than the threshold are selected. In my case, I set the threshold to mean*1.25.

I cannot understand why the output selector.threshold_ is not the same as selector.estimator_.coef_.mean()*1.25. I was expecting to get the same value, why is it not the case?

enter image description here

Moving forward, I wanted to do GridSearchCV to fine-tune my pipeline parameters. I usually do it this way:

from sklearn.model_selection import GridSearchCV

params = {}
params['gradientboostingclassifier__learning_rate'] = [0.05, 0.1, 0.2]
params['selectfrommodel__estimator__C'] = [0.1, 1, 10]
params['selectfrommodel__estimator__penalty']= ['l1', 'l2']
params['selectfrommodel__estimator__threshold']=['median', 'mean', '1.25*mean', '0.75*mean']

grid = GridSearchCV(pipe, params, cv=5, scoring='recall')
%time grid.fit(X_train, y_train);

Unfortunately, it looks like the threshold is not in the list of parameters (pipe.named_steps.selectfrommodel.estimator.get_params().keys()) so for GridSearchCV to work this line needs to be commented.

params['selectfrommodel__estimator__threshold']=['median', 'mean', '1.25*mean', '0.75*mean']

Is there a way to fine-tune the threshold?

gridsearchcv logistic-regression scikit-learn machine-learning python
2 Answers

Because the importance is based on the mean of the absolute values of the coefficients. If you do the mean on relative values, the average importance will be lower

I have built an example to demonstrate the behaviour:

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LogisticRegression
X = [[ 0.87, -1.34,  0.31 ],
     [-2.79, -0.02, -0.85 ],
     [-1.34, -0.48, -2.55 ],
     [ 1.92,  1.48,  0.65 ]]
y = [0, 1, 0, 1]
selector = SelectFromModel(estimator=LogisticRegression(), threshold="1.25*mean").fit(X, y)
print(selector.estimator_.coef_)
print(selector.threshold_) # 0.6905659148858644
# note here the absolute transformation before the mean
print(abs(selector.estimator_.coef_).mean()*1.25) # 0.6905659148858644

Also note that the feature importance is a consequence of your model training, not something that you can define a priori. That's the reason because you cannot fit a threshold, which is obtained only after your training

user_36533430

@Nikaido question was totally correct for the first part and abs() was missing. This means that abs(selector.estimator_.coef_).mean()*1.25 is equal to selector.threshold_

For the second part, it is indeed possible and the right way to do it is changing this line:

params['selectfrommodel__estimator__threshold']=['median', 'mean', '1.25*mean', '0.75*mean']

to this other line:

params['selectfrommodel__threshold']=['median', 'mean', '1.25*mean', '0.75*mean']

as threshold is a parameter from selectfrommodel and not from estimator, see below how to get the full list for both cases for further tuning of hyperparameters use these:

pipe.named_steps.selectfrommodel.get_params().keys() 
pipe.named_steps.selectfrommodel.estimator.get_params().keys()
user_77626460
Related