I have an interesting problem. I am working with a MULTICLASS problem (~90 classes), and have settled on using OneVsRestClassifier wrapper around a RandomForestClassifier.
When I call a .predict_proba() method on a fitted OVR(RF) object, the output looks like so:
clf = OneVsRestClassifier(BalancedRandomForestClassifier(random_state = random_state, n_jobs = n_jobs))
pipeline = Pipeline([('nt', nt), ('ros', ros), ('rf', clf)])
pipeline.fit(X_train, y_train)
...
pipeline.predict_proba(X_test.loc[1453].to_frame().T)
OUTPUT:
array([[0.01197937, 0.01422733, 0.00283992, 0.01277443, 0.00551339,
0.00156039, 0.00921673, 0.00887344, 0.01907149, 0.00405703,
0.00644963, 0.01254557, 0.02502873, 0.00187247, 0.01256638,
0.01255597, 0.00767093, 0.02044117, 0.01055867, 0.00691775,
0.01103719, 0.01184859, 0.00530534, 0.02320306, 0.00715701,
0.0043691 , 0.01152891, 0.00846774, 0.00894626, 0.01759085,
0.00957042, 0.01096484, 0.00995532, 0.01261618, 0.01168562,
0.020138 , 0.01998791, 0.01165094, 0.01651937, 0.0167304 ,
0.00729028, 0.00530534, 0.00702177, 0.005555 , 0.00421306,
0.01811098, 0.01156772, 0.00308958, 0.0063456 , 0.00644369,
0.01966097, 0.00312079, 0.00837412, 0.01342459, 0.01638414,
0.0090535 , 0.01673783, 0.01432442, 0.00865845, 0.01594723,
0.015812 , 0.00931035, 0.01319846, 0.02122136, 0.01262805,
0.01320614, 0.02954347, 0.02619382, 0.01372107, 0.00182446,
0.02115181, 0.00540937, 0.0320505 , 0.02377001, 0.00942478,
0.00674462, 0.00864172, 0.01466298, 0.00863938, 0.02129418,
0.01645349, 0.01493818, 0.00353689]])
What stands out to me about this output, is that all these probabilities are extremely low (and obviously add up to 1.0, as expected). There seems to be a really small margin of probability difference between correct and all other incorrect classes.
However, when I remove the OVR wrapper from the RandomForestClassifier, my output looks like such:
array([[0.00333333, 0.005 , 0. , 0.00166667, 0.005 ,
0. , 0. , 0. , 0.268375 , 0. ,
0. , 0.01766667, 0.0185 , 0. , 0.01666667,
0. , 0. , 0.02366667, 0. , 0. ,
0.01166667, 0.000625 , 0.005 , 0.0725 , 0. ,
0.009 , 0.01 , 0. , 0.00333333, 0.015 ,
0.02933333, 0.01 , 0. , 0. , 0.008 ,
0.04033333, 0.06833333, 0. , 0.04666667, 0. ,
0. , 0.0075 , 0.01 , 0. , 0. ,
0.01 , 0. , 0. , 0.02 , 0. ,
0.01 , 0. , 0. , 0.01 , 0. ,
0. , 0.0225 , 0. , 0.02 , 0.01 ,
0.0125 , 0. , 0. , 0. , 0.01 ,
0.0025 , 0.015 , 0.02 , 0.012 , 0. ,
0. , 0. , 0.03333333, 0.008 , 0.01 ,
0. , 0.02666667, 0. , 0. , 0.02783333,
0.0025 , 0.01 , 0. ]])
There is a clear standout class with a much higher probability than the rest.
My question is: am I missing something with the OVR approach? It is giving me better results than a non-OVR approach, but I would also expect it to product a stand-out probability, like to non-OVR one.
The way I understand it, what is happening with the OVR wrapper each class (out of ~90) gets 2 classifiers (Yes/No) and both Y/N cases get a mean value of votes from all DecisionTrees in that RF. Then predict_proba() should produce the higher one of these Y/N votes?