I'm currently using xgb.train(...) which returns a booster but I'd like to use RFE to select the best 100 features. The returned booster cannot be used in RFE as it's not a sklearn estimator. XGBClassifier is the sklearn api into the xgboost library, however, I am not able to get the same results as with the xgb.train(...) method (10% worse on roc-auc). I've tried the sklearn boosters but they're not able to get similar results either. I've also tried to wrap the xgb.train(...) method in a class to add sklearn estimator methods but there's just too many to change. Is there some way to use the xgb.train(...) along with RFE from sklearn?
How to use RFE with xgboost Booster?
Viewed 1378
1 Answers
For this kind of problem, I created shap-hypetune: a python package for simultaneous Hyperparameters Tuning and Features Selection for Gradient Boosting Models
In your case, this enables you to perform RFE with XGBClassifier in a very simple and intuitive way:
from shaphypetune import BoostRFE
model = BoostRFE(XGBClassifier(), min_features_to_select=1, step=1)
model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], early_stopping_rounds=6, verbose=0)
pred = model.predict(X_test)
As you can see, you can use all the fitting options available in the standard XGB API, like early_stopping_rounds or custom metrics, to customize the training process.
You can use shap-hypetune also to compute parameter tuning (also simultaneously with feature selection) or to compute feature selection with RFE or Boruta using SHAP feature importance. Full example available here
Related