Gradient Boosting vs Random forest

user_7739030 0
Viewed 5727

According to my understanding, RF selects features randomly and hence is hard to overfit. But, in sklearn Gradient boosting also offers the option of max_features which can help to prevent overfitting. So, why would anyone use Random forest? Can anyone explain when to use Gradient boosting vs Random forest based on the given data?

Any help is highly appreciated.

xgboost machine-learning random-forest
3 Answers

According to my personal experience, Random Forest could be a better choice when..

  1. You train a model on small data set.
  2. Your data set has few features to learn.
  3. Your data set has low Y flag count or you try to predict a situation that has low chance to occur or rarely occurs.

In these situations, Gradient Boosting algorithms like XGBoost and Light GBM can overfit (though their parameters are tuned) while simple algorithms like Random Forest or even Logistic Regression may perform better. To illustrate, for XGboost and Ligh GBM, ROC AUC from test set may be higher in comparison with Random Forest but shows too high difference with ROC AUC from train set.

Despite the sharp prediction form Gradient Boosting algorithms, in some cases, Random Forest take advantage of model stability from begging methodology (selecting randomly) and outperform XGBoost and Light GBM. However, Gradient Boosting algorithms perform better in general situations.

user_92344860

I think that's also true. I have also read on this page How Random Forest Works There explains the advantages of random forest. like this :

  1. For applications in classification problems, Random Forest algorithm will avoid the overfitting problem
  2. For both classification and regression task, the same random forest algorithm can be used
  3. The Random Forest algorithm can be used for identifying the most important features from the training dataset, in other words, feature engineering.
user_97760850
Related