Scikit learn custom scoring function - Specificity

Question

I'm trying to do a random grid search on randomforestclassifier.

# Instantiate a RandomForestClassifier 
RFC = RandomForestClassifier()

# Instantiate the RandomizedSearchCV object: RFC
rand_search3 = RandomizedSearchCV(RFC, param_grid, n_iter=10, cv=5,n_jobs=-1, verbose=1, scoring = "f1_macro")

# Fit it to the data
rand_search3.fit(X_train_transformed,y_train)

I'm trying to get the best model by assessing specificity. Went through the documentation for custom scoring. Also looked at lots of posts that are related. I have came up with 2 ways for the specificity. 1 :

from sklearn.metrics import make_scorer

def my_custom_func(y_true, y_pred):
     cm = confusion_matrix(y_true, y_pred)
     return cm[1][1] / (cm[1][0] + cm[1][1])

Specificity_score = make_scorer(my_custom_func, greater_is_better=True)

2:

from sklearn.metrics import recall_score
from sklearn.metrics import make_scorer

specificity = make_scorer(recall_score, pos_label=0, greater_is_better=True)
specificity

When I try using the custom function for the scoring,

rand_search3 = RandomizedSearchCV(RFC, param_grid, n_iter=10, cv=5,n_jobs=-1, verbose=1, scoring = Specificity_score)
rand_search3 = RandomizedSearchCV(RFC, param_grid, n_iter=10, cv=5,n_jobs=-1, verbose=1, scoring = specificity)

both failed with the same error message.

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_10548/1204393696.py in 
     20 
     21 # Fit it to the data
---> 22 rand_search3.fit(X_train_transformed,y_train)
     23 

~\anaconda3\lib\site-packages\sklearn\utils\validation.py in inner_f(*args, **kwargs)
     61         @wraps(f)
     62         def inner_f(*args, **kwargs):
---> 63             extra_args = len(args) - len(all_args)
     64             if extra_args <= 0:
     65                 return f(*args, **kwargs)

~\anaconda3\lib\site-packages\sklearn\model_selection\_search.py in fit(self, X, y, groups, **fit_params)
    839                     delayed(_fit_and_score)(
    840                         clone(base_estimator),
--> 841                         X,
    842                         y,
    843                         train=train,

~\anaconda3\lib\site-packages\sklearn\model_selection\_search.py in _run_search(self, evaluate_candidates)
   1631         Mean cross-validated score of the best_estimator.
   1632 
-> 1633         For multi-metric evaluation, this is not available if ``refit`` is
   1634         ``False``. See ``refit`` parameter for more information.
   1635 

~\anaconda3\lib\site-packages\sklearn\model_selection\_search.py in evaluate_candidates(candidate_params, cv, more_results)
    825             def evaluate_candidates(candidate_params, cv=None, more_results=None):
    826                 cv = cv or cv_orig
--> 827                 candidate_params = list(candidate_params)
    828                 n_candidates = len(candidate_params)
    829 

~\anaconda3\lib\site-packages\sklearn\model_selection\_validation.py in _insert_error_scores(results, error_score)
    293 
    294     results = _aggregate_score_dicts(results)
--> 295 
    296     ret = {}
    297     ret["fit_time"] = results["fit_time"]

KeyError: 'fit_failed'

Any solutions?

Scikit learn custom scoring function - Specificity

0 Answers