sklearn RFE with GaussianProcessClassifier

user_9998989 0
Viewed 587

I'm working with a recursive feature elimination (RFE) pipeline:

from sklearn.feature_selection import RFE

c_GAUSS = GaussianProcessClassifier()
rfe = RFE(estimator=c_GAUSS)

However, I get an error:

ValueError: when `importance_getter=='auto'`, the underlying 
estimator GaussianProcessClassifier should have `coef_` or 
`feature_importances_` attribute. Either pass a fitted estimator to 
feature selector or call fit before calling transform.

With SVC, I just solved this problem by putting a coefficient in the SVC with c_SVC = SVC(C = 1).

Is there any possible workaround here, since GPC doesn't have coefficients?

scikit-learn python
2 Answers

According to the RFE documentation, the estimator requires information about feature importances (e.g. estimator.coef_, estimator.feature_importances_). Unfortunately, the GPC estimator does not come with this attribute, due to the way classification is done in GPs.

According to an answer on Cross Validated, one approach to determine feature importance in GP classification would be to determine the classification error or other metric you're using by successively leaving out one feature at a time. This is time-intensive and not done by the GP by default. However, this would require implementing your own RFE function (not too difficult).

user_37916400

I'm afraid there's not. As the documentation of RFE states, the estimator is expected "to assign weights to features". A compatible estimator has to expose those weights either through the coef_ or feature_importances_ attribute. Have a look at the implementation of RFE, it's pretty straightforward.

I'm not very familiar with GaussianProcessClassifier, however, it seems that this type of classifier does not perform any feature scoring. If you think that there's a natural way to score features for a GPC after training, you could extend the GaussianProcessClassifier yourself:

from sklearn.utils.validation import check_X_y
from sklearn.utils.estimator_checks import check_estimator
class WeightedGaussianProcessClassifier(GaussianProcessClassifier):
    def fit(self, X, y):
        X, y = check_X_y(X, y)
        super().fit(X,y)
        # Implement here your weight
        self.feature_importances_ = ...
        return self
check_estimator(WeightedGaussianProcessClassifier)

Follow the instructions here on how to develop your own estimator. This way, you make the estimator compatible with the RFE class.

user_33889620
Related