How to perform a random gridsearch within selecting features from a model?

user_8831033 0
Viewed 785

I am looking to select features based on feature importance of either random forest, gradient boosting and extreme gradient boosting. I am trying to fit my models with using a ranomdized gridsearch to get the best model's feature importances, but it gives me an error I don't understand, here is my code:

gbr = GradientBoostingRegressor(random_state=seed)
gbr_params = {
    "learning_rate": [0.001, 0.01, 0.1],
    "min_samples_split": [50, 100],
    "min_samples_leaf": [50, 100],
    "max_depth":[5, 10, 20]}

xgbr = xgboost.XGBRegressor(random_state=seed) 
xgbr_params = {  
    "learning_rate": [0.001, 0.01, 0.1],
    "min_samples_leaf": [50, 100],
    "max_depth":[5, 10, 20],
    'reg_alpha': [1.1, 1.2, 1.3],
    'reg_lambda': [1.1, 1.2, 1.3]}

rfr = RandomForestRegressor(random_state=seed)
rfr_params={'n_estimators':[100, 500, 1000], 
             'max_features':[10,14,18],
             'min_samples_split': [50, 100],
             'min_samples_leaf': [50, 100],} 

fs_xgbr = dcv.RandomizedSearchCV(xgbr, xgbr_params, cv=5, iid=False, n_jobs=-1)
fs_gbr = dcv.RandomizedSearchCV(gbr, gbr_params, cv=5,iid=False, n_jobs=-1)
fs_rfr = dcv.RandomizedSearchCV(rfr, rfr_params, cv=5,iid=False, n_jobs=-1)

fs_rfr.fit(X, Y)
model = SelectFromModel(fs_rfr, prefit=True)
X_rfr = model.transform(X)
print('rfr', X_rfr.shape)

At the line of X_rfr = model.transform(X) it gives this error:

ValueError: The underlying estimator RandomizedSearchCV has no `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to SelectFromModel or call fit before calling transform.

I am not a programmer and haven't found any solution elsewhere to solve this, is it not possible to take feature_importances_ of the model with it's best parameters decided by the randomizedsearch?

grid-search gridsearchcv scikit-learn machine-learning python
1 Answers

Instead of passing to SelectFromModel fs_rfr, which is an object of RandomizedSearchCV type, pass best estimator, like fs_rfr.best_estimator_

Proof

import xgboost
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import RandomizedSearchCV
from sklearn.feature_selection import SelectFromModel

seed=42

gbr = GradientBoostingRegressor(random_state=seed)
gbr_params = {
    "learning_rate": [0.001, 0.01, 0.1],
    "min_samples_split": [50, 100],
    "min_samples_leaf": [50, 100],
    "max_depth":[5, 10, 20]}

xgbr = xgboost.XGBRegressor(random_state=seed) 
xgbr_params = {  
    "learning_rate": [0.001, 0.01, 0.1],
    "min_samples_leaf": [50, 100],
    "max_depth":[5, 10, 20],
    'reg_alpha': [1.1, 1.2, 1.3],
    'reg_lambda': [1.1, 1.2, 1.3]}

rfr = RandomForestRegressor(random_state=seed)
rfr_params={'n_estimators':[100, 500, 1000], 
             'max_features':[10,14,18],
             'min_samples_split': [50, 100],
             'min_samples_leaf': [50, 100],} 

fs_xgbr = RandomizedSearchCV(xgbr, xgbr_params, cv=5, iid=False, n_jobs=-1)
fs_gbr = RandomizedSearchCV(gbr, gbr_params, cv=5,iid=False, n_jobs=-1)
fs_rfr = RandomizedSearchCV(rfr, rfr_params, cv=5,iid=False, n_jobs=-1)

X, y = make_regression(1000,10)

fs_xgbr.fit(X, y)
fs_gbr.fit(X, y)
fs_rfr.fit(X, y)

model = SelectFromModel(fs_rfr.best_estimator_, prefit=True)
X_rfr = model.transform(X)
print('rfr', X_rfr.shape)

model = SelectFromModel(fs_xgbr.best_estimator_, prefit=True)
X_xgbr = model.transform(X)
print('xgbr', X_xgbr.shape)

model = SelectFromModel(fs_gbr.best_estimator_, prefit=True)
X_gbr = model.transform(X)
print('gbr', X_gbr.shape)

rfr (1000, 3)
xgbr (1000, 3)
gbr (1000, 4)
user_43170580
Related