Why the voting classifier has less accuracy than one of the individual predictors that made it

user_13093694 0
Viewed 539

I have a simple question concerning the votting classifier. As I understood, the voting classifier should have the highest accuracy than those individual predictors which built it (the wisdom of the crowd). Here is the code

from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.svm import SVC


# import dataset
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)

# split the dataset into train/test sets
X_train, X_test, y_train, y_test = train_test_split(X, y)


rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)  
log_clf = LogisticRegression(solver='liblinear', random_state=42) 
svm_clf = SVC(gamma='auto', random_state=42)   


voting_clf = VotingClassifier(
    estimators= [('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='hard')                                          
                      

voting_clf = voting_clf.fit(X_train, y_train)

predictors_list= [log_clf, rnd_clf, svm_clf, voting_clf]

for clf in predictors_list:
    
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    accuracy = accuracy_score(y_pred, y_test)
    
    print(clf.__class__.__name__, accuracy)

what I get as a accuracy is as follows:

LogisticRegression 0.776 RandomForestClassifier 0.88 SVC 0.864 VotingClassifier 0.864

As you can see for this run the Random Forest predictor has a slightly better accuray than the VotingClassifier!

Any explanation for this?

Many thanks in Advance

Fethi

voting classification python
1 Answers

Let's take a look at the voting parameter you passed 'hard' documentation says:

If ‘hard’, uses predicted class labels for majority rule voting. Else if ‘soft’, predicts the class label based on the argmax of the sums of the predicted probabilities, which is recommended for an ensemble of well-calibrated classifiers.

So maybe, the prediction of ‍‍‍‍LogisticRegression and your SVC(SVM) are the same and wrong for some cases this makes your majority vote wrong for those cases.

you can use voting='soft' or assign weight as prior for prediction of each model, this way you make your prediction a little bit immune to the wrong prediction of bad models, and relay more on your best models.

user_103215310
Related