How do I solve overfitting in random forest of Python sklearn?

user_1118236 0
Viewed 62271

I am using RandomForestClassifier implemented in python sklearn package to build a binary classification model. The below is the results of cross validations:

Fold 1 : Train: 164  Test: 40
Train Accuracy: 0.914634146341
Test Accuracy: 0.55

Fold 2 : Train: 163  Test: 41
Train Accuracy: 0.871165644172
Test Accuracy: 0.707317073171

Fold 3 : Train: 163  Test: 41
Train Accuracy: 0.889570552147
Test Accuracy: 0.585365853659

Fold 4 : Train: 163  Test: 41
Train Accuracy: 0.871165644172
Test Accuracy: 0.756097560976

Fold 5 : Train: 163  Test: 41
Train Accuracy: 0.883435582822
Test Accuracy: 0.512195121951

I am using "Price" feature to predict "quality" which is a ordinal value. In each cross validation, there are 163 training examples and 41 test examples.

Apparently, overfitting occurs here. So is there any parameters provided by sklearn can be used to overcome this problem? I found some parameters here, e.g. min_samples_split and min_sample_leaf, but I do not quite understand how to tune them.

Thanks in advance!

scikit-learn decision-tree machine-learning random-forest python
2 Answers

Adding this late comment in case it helps others.

In addition to the parameters mentioned above (n_estimators, max_features, max_depth, and min_samples_leaf) consider setting 'min_impurity_decrease'.

Doing this manually is cumbersome. So use sklearn.model_selection.GridSearchCV to test a range of parameters (parameter grid) and find the optimal parameters.

You can use 'gini' or 'entropy' for the Criterion, however, I recommend sticking with 'gini', the default. In the majority of cases, they produce the same result but 'entropy' is more computational expensive to compute.

Max depth works well and is an intuitive way to stop a tree from growing, however, just because a node is less than the max depth doesn't always mean it should split. If the information gained from splitting only addresses a single/few misclassification(s) then splitting that node may be supporting overfitting. You may or may not find this parameter useful, depending on the size of your dataset and/or your feature space size and complexity, but it is worth considering while tuning your parameters.

user_172147530
Related