I want to use nested cross-validation with Grid search for a 2-class classification problem, using the roc_auc function as a scorer. I also want to print the classification matrix, so I have tried to create a simple custom scorer function which prints out a classification report. However, I get a different nested_score with the 2 functions. Here is an example using the breast cancer dataset adapted from sklearn's example (https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html):
from sklearn.datasets import load_breast_cancer
from matplotlib import pyplot as plt
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, cross_val_score, KFold
from sklearn import metrics
import numpy as np
def classification_report_with_roc_score(y_true, y_pred):
print (classification_report(y_true, y_pred)) # print classification report
fpr, tpr, thresholds = metrics.roc_curve(y_true, y_pred)
roc_auc = metrics.auc(fpr, tpr)
return roc_auc # return auc score
NUM_TRIALS = 1
breast_cancer = load_breast_cancer()
X_cancer = breast_cancer.data
y_cancer = breast_cancer.target
p_grid = {"C": [1, 10, 100],
"gamma": [.01, .1]}
svm = SVC(kernel="rbf")
for i in range(NUM_TRIALS):
inner_cv = KFold(n_splits=4, shuffle=True, random_state=i)
outer_cv = KFold(n_splits=4, shuffle=True, random_state=i)
clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv)
nested_score = cross_val_score(clf, X=X_cancer, y=y_cancer, scoring = 'roc_auc', cv=outer_cv)
print('nested_score', nested_score)
custom_nested_score = cross_val_score(clf, X=X_cancer, y=y_cancer, scoring =
make_scorer(classification_report_with_roc_score), cv=outer_cv)
print('nested_score_custom', custom_nested_score)
The result is
nested_score [0.9836478 0.97074468 0.97853535 0.98266254]
nested_score_custom [0.92672956 0.92176418 0.88110269 0.89174407]
I was expecting them to be the same. Can someone please provide suggestions for why the results are different and what has gone wrong with the classification_report_with_roc_score() function? Thank you.