Does scikit-learn use One-Vs-Rest by default in multi-class classification?

user_4956494 0
Viewed 5391

I am dealing with a multi-class problem (4 classes) and I am trying to solve it with scikit-learn in Python.

I saw that I have three options:

  1. I simply instantiate a classifier, then I fit with train and evaluate with test;

    classifier = sv.LinearSVC(random_state=123)
classifier.fit(Xtrain, ytrain)
classifier.score(Xtest, ytest)

  2. I "encapsulate" the instantiated classifier in a OneVsRest object, generating a new classifier that I use for train and test;

    classifier = OneVsRestClassifier(svm.LinearSVC(random_state=123))
classifier.fit(Xtrain, ytrain)
classifier.score(Xtest, ytest)

  3. I "encapsulate" the instantiated classifier in a OneVsOne object, generating a new classifier that I use for train and test.

    classifier = OneVsOneClassifier(svm.LinearSVC(random_state=123))
classifier.fit(Xtrain, ytrain)
classifier.score(Xtest, ytest)

I understand the difference between OneVsRest and OneVsOne, but I cannot understand what I am doing in the first scenario where I do not explicitly pick up any of these two options. What does scikit-learn do in that case? Does it implicitly use OneVsRest?

Any clarification on the matter would be highly appreciated.

Best, MR

Edit: Just to make things clear, I am not specifically interested in the case of SVMs. For example, what about RandomForest?

multiclass-classification scikit-learn machine-learning python
1 Answers

Updated answer: As clarified in the comments and edits, the question is more about the general setting of sklearn, and less about the specific case of LinearSVC which is explained below.

The main difference here is that some of the classifiers you can use have "built-in multiclass classification support", i.e. it is possible for that algorithm to discern between more than two classes by default. One example for this would for example be a Random Forest, or a Multi-Layer Perceptron (MLP) with multiple output nodes.

In these cases, having a OneVs object is not required at all, since you are already solving your task. In fact, using such a strategie might even decreaes your performance, since you are "hiding" potential correlations from the algorithm, by letting it only decide between single binary instances.

On the other hand, algorithms like SVC or LinearSVC only support binary classification. So, to extend these classes of (well-performing) algorithms, we instead have to rely on the reduction to a binary classification task, from our initial multiclass classification task.

As far as I am aware of, the most complete overview can be found here: If you scroll down a little bit, you can see which one of the algorithms is inherently multiclass, or uses either one of the strategies by default.
Note that all of the listed algorithms under OVO actually now employ a OVR strategy by default! This seems to be slightly outdated information in that regard.

Initial answer:

This is a question that can easily be answered by looking at the relevant scikit-learn documentation.
Generally, the expectation on Stackoverflow is that you have at least done some form of research on your own, so please consider looking into existing documentation first.

multi_class : string, ‘ovr’ or ‘crammer_singer’ (default=’ovr’)

Determines the multi-class strategy if y contains more than two classes. "ovr" trains n_classes one-vs-rest classifiers, while "crammer_singer" optimizes a joint objective over all classes. While crammer_singer is interesting from a theoretical perspective as it is consistent, it is seldom used in practice as it rarely leads to better accuracy and is more expensive to compute. If "crammer_singer" is chosen, the options loss, penalty and dual will be ignored.

So, clearly, it uses one-vs-rest.

The same holds by the way for the "regular" SVC.

user_36072030
Related