Why recursive feature elimination procedure does not get rid of useless predictors?

user_8856229 0
Viewed 618

I am trying to select the best predictors of a variable y

x1 and x3 are predictors of y, x2 is correlated to x1 and x4 is a dummy variable.

library(randomForest);library(caret)
set.seed(123)
x1<-rnorm(1000,sd=.3,mean=-2)
x3<-rnorm(1000,sd=1,mean=.3)
x2<-jitter(x1,amount=1)
x4<-rnorm(1000,sd=4,mean=3)

y<-jitter(3*x1+jitter(x3,amount=2),amount=2)
varImpPlot(randomForest(y~x1+x2+x3+x4,importance=T))

enter image description here

ctrl <- rfeControl(functions = rfFuncs,number=3)
x<-data.frame(x1,x2,x3,x4)
rfe(x,y,rfeControl=ctrl,sizes=1:4,method="rf")

#...
#The top 4 variables (out of 4):
#x3, x1, x2, x4

cor(x)
#             x1          x2         x3          x4
# x1  1.00000000  0.45351111 0.08647944 -0.02470308
# x2  0.45351111  1.00000000 0.03927750 -0.08157149
# x3  0.08647944  0.03927750 1.00000000  0.04357772
# x4 -0.02470308 -0.08157149 0.04357772  1.00000000
  • Why does recursive feature elimination procedure tell me to keep all predictors even if it is very clear when looking at variable importance that x2 and x4 are useless ?
rfe feature-selection r-caret random-forest r
1 Answers

The importance you visualize with varImpPlot, should not be used has a stand-alone method to remove non important variables.

First of all, continous variables and categorical variables with more labels will have a higher importance. That can be misleading.

Second, correlated predictors can have low variable importance. That's the opposite of what you want, sometimes.

Also the importance doesn't tell you how the predictors together are related with the response.

I suggest to use the permutation method, where you check for importance by re-estimating the model after you permute one variable (ex: sample(x4)), check how the performance moves, comparing for example mse (before and after permutation).

The simple idea is that if a variable is useless the performance won't change much.

Lastly, here are some useful readings.

Link1

Link2

user_56582700
Related