Custom make_scorer for roc_auc score gives different result compared to scoring = 'roc_auc' in 2 class classification

Viewed 219

I want to use nested cross-validation with Grid search for a 2-class classification problem, using the roc_auc function as a scorer. I also want to print the classification matrix, so I have tried to create a simple custom scorer function which prints out a classification report. However, I get a different nested_score with the 2 functions. Here is an example using the breast cancer dataset adapted from sklearn's example (https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html):

from sklearn.datasets import load_breast_cancer
from matplotlib import pyplot as plt
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, cross_val_score, KFold
from sklearn import metrics
import numpy as np

def classification_report_with_roc_score(y_true, y_pred):
    print (classification_report(y_true, y_pred)) # print classification report
    fpr, tpr, thresholds = metrics.roc_curve(y_true, y_pred)
    roc_auc = metrics.auc(fpr, tpr)
    return roc_auc # return auc score

NUM_TRIALS = 1

breast_cancer = load_breast_cancer()
X_cancer = breast_cancer.data
y_cancer = breast_cancer.target

p_grid = {"C": [1, 10, 100],
      "gamma": [.01, .1]}
svm = SVC(kernel="rbf")

for i in range(NUM_TRIALS):
    inner_cv = KFold(n_splits=4, shuffle=True, random_state=i)
    outer_cv = KFold(n_splits=4, shuffle=True, random_state=i)
    clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv)
    nested_score = cross_val_score(clf, X=X_cancer, y=y_cancer, scoring = 'roc_auc', cv=outer_cv)   
    print('nested_score', nested_score)
    custom_nested_score = cross_val_score(clf, X=X_cancer, y=y_cancer, scoring = 
                        make_scorer(classification_report_with_roc_score), cv=outer_cv)
    print('nested_score_custom', custom_nested_score)

The result is

nested_score [0.9836478  0.97074468 0.97853535 0.98266254]
nested_score_custom [0.92672956 0.92176418 0.88110269 0.89174407]

I was expecting them to be the same. Can someone please provide suggestions for why the results are different and what has gone wrong with the classification_report_with_roc_score() function? Thank you.

1 Answers

As your scorer needs probabilties to be calculated you have to set 'needs_proba' argument of the make_scorer function to True.

custom_nested_score = cross_val_score(clf, X=X_cancer, y=y_cancer, scoring = 
                    metrics.make_scorer(classification_report_with_roc_score, needs_proba=True), cv=outer_cv)

Secondly, you also have to set 'probability=True' when you initialize you SVC:

svm = SVC(kernel="rbf", probability=True)

Doing so, i got following results running your code:

nested_score [0.91278826 0.94326241 0.94760101 0.94097007]
nested_score_custom [0.91278826 0.94326241 0.94760101 0.94097007]
Related