Using Sklearn random forest for feature selection does not give me expected outcome when having categorical data

user_14404077 0
Viewed 247

I would like to use SKlearn random forest feature selection function to understand what are the key factors impacting my independent variable (TN pollutant concentration). I have one categorical variables - Climate type, with five types of climate (Temperature-hot, temperature-dry, temperatre warm, tropical and arid), I knew that the climate type has big impact on my independent variable, however, when I used the One hot encoding approach (through pandas get_dummies)，I found these climate types (that become five variables with false/true after one hot encoding) were least important, which is not true.

As shown here, the climate variables have the least feature importance score:

My question is that whether the feature selection of random forest is still useful when dealing with categorical variables? and if I did something wrong about it?

This is part of my code:

  model = RandomForestRegressor(n_estimators=100, bootstrap = True,max_features = 'sqrt')

  model.fit(x_train, y_train)

  fi = pd.DataFrame({'feature': list(x_train),'importance': 
  model.feature_importances_}).sort_values('importance', ascending = False)

  plt.bar(fi['feature'],fi['importance'])
feature-selection scikit-learn random-forest python
1 Answers

It all depends how the feature-importance is calculated. If the feature-importance of a feature is a function of how many times it is splitting a node, then it is difficult to compare numeric and categorical values, since numerical featues can (and often are) be split multiple times in a tree, where categories only are split once.

I am not completely sure, but I think that the feature importance in sklearn is some function of the amount of splits of a feature, thus the mis-leading importance.

user_62249750
Related