Too good to be true results using sklearn's RandomForestRegressor compared with R's RandomForest?

JBSacristan 0
Viewed 75

I’m currently working with random forests both in Python (RandomForestRegressor() from sklearn) and R (randomforest() function from the RandomForest package) and I wonder if someone has experienced consistently better results in Python.

I’m attaching both the code and the results obtained using the mtcars dataset. This is not the dataset I’m interested in right now, but I constantly keep having better results (sometimes in the order of R-squared of 0.50 versus 0.93) in Python no matter the dataset I use, and this makes me suspicious about it.

Python

inputfile = r'/PathToFile/mtcars.csv'
myData = pd.read_csv(inputfile)

# Get the features and labels for random forest regressor
df_x = data[["cyl","disp","hp","drat","wt","qsec","vs","am","gear","carb"]]
df_reference = data['mpg']

np.random.seed(1)
clf = RandomForestRegressor(n_estimators=1000, max_features=7)

# Train the model
clf.fit(df_x, df_reference)
    
# Create prediction
df_predicted = clf.predict(df_x)

R2 = metrics.r2_score(df_reference, df_predicted)
MAE = metrics.mean_absolute_error(df_reference, df_predicted)
RMSE = math.sqrt(MSE)

print(R2);print(MAE);print(RMSE)

R

library(Metrics)
library(randomForest)
rm(list=ls())

set.seed(1)
data(mtcars)
rf.fit <- randomForest(mpg ~ ., data=mtcars, ntree=1000, mtry=7)

summary(lm(rf.fit$predicted~mtcars$mpg))$r.squared
mae(mtcars$mpg,rf.fit$predicted)
rmse(mtcars$mpg,rf.fit$predicted)

The results are as follows:

Python

r2 = 0.97
MAE = 0.68
RMSE = 0.85

R

r2 = 0.84
MAE = 1.84
RMSE = 2.32

As you can see, the results are quite better in Python. In this case, no great differences, but in the relatively bigger datasets I'm working with (several hundreds of rows) I always get better results using sklearn's approach.

Is there any reason for this to always be the case? Am I missing something?

In some of the datasets I have been using, the r2 have different values of more than 0.3, which seems to be too much to be attributed to the random nature of RF.

As far as I understand the two parameters I'm setting in both approaches are the ones affecting the most the results, but I guess I'm missing something here.

I've seen relatively similar posts like this, but nothing explaining this.

random-forest machine-learning scikit-learn r python
1 Answers

The default for nodesize in R's randomForest is 5 for regression (1 for classification), but the corresponding min_samples_leaf in sklearn is 1. This will lead to more-overfit individual trees in sklearn, but in the aggregation across bags it may be better or worse. Note that you're reporting the training set metrics, so those are optimistically biased anyway; it'd be interesting to see how the two models perform on some out-of-sample data.

Ben Reiniger0
Related