I wanted to get a much faster random forest classifier than the one from Weka, I first tried the C++ Shark implementation (results: few speed improvement, drop in correctly classifed instances) and then tested Python Scikit-learn. I read on many websites and papers that Weka performs badly compared to Scikit, WiseRF...
After my first try with a forest of 100 trees:
Training time: Weka ~ 170s VS Scikit ~ 31s
Prediction results on the same test set: Weka ~ 90% correctly classified VS Scikit score ~ 45% !!!
=> Scikit RF runs fast but classify very badly on this first try.
I tuned the parameters of Scikit RandomForestClassifier and managed to get a score close to 70% but the speed of scikit dropped nearly down to Weka performance (bootstrap=False, min_samples_leaf=3, min_samples_split=1, criterion='entropy', max_features=40, max_depth=6). I do have many missing values and scikit does not handle them out of the box so I tried many different strategies (all strategies of Imputer, skip instances with missing values, replace with 0 or extreme values) and reached 75%.
So at this stage Scikit RandomForestClassifier performs at 75% (compared to 90% with weka) and build the model in 78s (using 6 cores vs 170s with only 1 core with Weka). I am very surprised with those results. I tested ExtraTrees which performs very well in terms of speed but still reach an average of 75% correct classification.
Do you have any idea what I am missing ?
My data: ~100 features, ~100 000 instances, missing values, classification prediction (price forecast).