I ran a nested-cross validation procedure on my data set. Now I would like to assess the significance of the average performance of all outer models. Is there any package that allows me to do this? I came across sklearn.model_selection.permutation_test_score but it seems that this function only takes one classifier as input and I also don't know how I would have to implement this function for my problem. I am looking for a way to assess the significance of the average classification performance of all surrogate models that are returned by cross_validate.
Example:
from sklearn.datasets import load_iris
from sklearn.model_selection import KFold
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import cross_validate
from sklearn.svm import SVC
import numpy as np
import matplotlib.pyplot as plt
# get iris data set
iris = load_iris()
X = iris.data
y = iris.target
# set random seed
random_seed = 42
# use support vector machine
svr = SVC()
# optimize C-Parameter
p_grid = {"C": [1, 10, 100]}
# Define cross-validation strategies
inner_cv = KFold(shuffle=True,random_state=random_seed)
outer_cv = KFold(shuffle=True,random_state=random_seed)
# define inner cv
grid = GridSearchCV(svr,param_grid=p_grid,cv=inner_cv)
# define outer cv
scores = cross_validate(grid,X,y,cv=outer_cv,return_train_score=True,return_estimator=True)
# get average performance across all surrogate models
avg_test_score = np.mean(scores['test_score'])
# plot results
x = np.arange(len(scores['estimator']))
width = 0.35 # the width of the bars
fig, ax = plt.subplots()
train_scores = ax.bar(x - width/2,height=scores['train_score'],width=width,label='Train Score')
test_scores = ax.bar(x + width/2,height=scores['test_score'],width=width,label='Test Score')
ax.axhline(y=avg_test_score, xmin=0.0, xmax=1.0, color='orange')
ax.set_ylabel('Score')
ax.set_xticks(x)
ax.set_xticklabels(['Outer Fold 1','Outer Fold 2','Outer Fold 3','Outer Fold 4','Outer Fold 5'])
ax.legend(bbox_to_anchor=(1,0.5),loc='center left')
In this example the average performance across all five surrogate models is avg_test_score≈0.97. Now is there a way to assess the significance of this average value by repeating the whole procedure n times, shuffling the labels, and obtain the frequency of this value or a higher one?
times
