Cannot use XGBRegressor with sklearn RFE

user_11267180 0
Viewed 920

HELP, this simple code yeilds a weird error.

from sklearn.feature_selection import RFECV
from xgboost.sklearn import XGBRegressor
import sklearn.metrics
from sklearn.metrics import mean_absolute_error

estimator = XGBRegressor()
selector = RFECV(estimator, step=1, min_features_to_select=1, cv=10, scoring='neg_mean_absolute_error')
selector = selector.fit(x, y.values.flatten())

My regressor is working smoothly already, but selector.fit does not. I get the same for either RFE or RFECV:

~/miniconda2/envs/py3/lib/python3.6/site-packages/xgboost/sklearn.py in coef_(self)
    714                                  .format(self.booster))
    715         b = self.get_booster()
--> 716         coef = np.array(json.loads(b.get_dump(dump_format='json')[0])['weight'])
    717         # Logic for multiclass classification
    718         n_classes = getattr(self, 'n_classes_', None)

KeyError: 'weight'

Thanks in advance.

feature-selection xgboost scikit-learn python
2 Answers

I met the same issue in xgboost Version: 1.0.2. Downgrading to version 0.90 as follows resolved the issue.

pip show xgboost
pip uninstall xgboost
pip install --upgrade xgboost==0.90
pip show xgboost

In version 0.90 I got following warning, the error may have to do with using (default) reg:linear as the metric. If you downgrade to v 0.90, you do not need to do anything and use your existing code (if it used to work before the upgrade).

WARNING: C:/Jenkins/workspace/xgboost-win64_release_0.90/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror

user_30140620

You could try to set the booster option in XGBRegressor:

estimator = XGBRegressor(booster='gbtree')

There are basically two types of boosters in XGBoost: linear and tree (https://xgboost.readthedocs.io/en/latest/parameter.html). There are some attributes that are exclusive to each one of them. For example, the coef_ property that gave you the error, is only defined for linear learners (https://xgboost.readthedocs.io/en/latest/python/python_api.html), while feature_importances_ are only defined for tree based ones.

user_149490550
Related