How to make custom scoring metric for cross_validate in scikit learn?

user_6227592 0
Viewed 955

I want to develop pr_auc as the scoring metric for cross_validate(). So I followed Scikit Learn's user guide: https://scikit-learn.org/stable/modules/model_evaluation.html#scoring

My code is shown below:

from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate
from xgboost import XGBClassifier
from sklearn.metrics import auc, make_scorer

def cus_pr_auc(x, y):
    score=auc(x, y)
    return score

X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
    n_clusters_per_class=2, weights=[0.9], flip_y=0, random_state=7)

model = XGBClassifier(scale_pos_weight=9)

scores = cross_validate(model, X, y, scoring=make_scorer(cus_pr_auc, greater_is_better=True), cv=3, n_jobs=-1)

However, I got the following error message:

ValueError: x is neither increasing nor decreasing : [1 1 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 1 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1].

How do I fix the code?

cross-validation scikit-learn python
1 Answers

Change your metric from auc to roc_auc_score as below and you're fine to go:

from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate
from xgboost import XGBClassifier
from sklearn.metrics import roc_auc_score, make_scorer

def cus_pr_auc(x, y):
    score=roc_auc_score(x, y)
    return score

X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
    n_clusters_per_class=2, weights=[0.9], flip_y=0, random_state=7)

model = XGBClassifier(scale_pos_weight=9)

scores = cross_validate(model, X, y, scoring=make_scorer(cus_pr_auc, greater_is_better=True), cv=3, n_jobs=-1)

scores
{'fit_time': array([0.0627017, 0.0569284, 0.046772 ]),
 'score_time': array([0.00534487, 0.00616908, 0.00347471]),
 'test_score': array([0.90244639, 0.90242424, 0.94969697])}

Your auc is not working because, according to docs:

x coordinates. These must be either monotonic increasing or monotonic decreasing.

Hence your error (see here as well, for concrete source of your error message).

Generally auc is a low level scoring metric which is used as auc(fpr,tpr).

user_43170580
Related