GridSearchCV optimising multiple metrics (scorer)

Carolyn 0
Viewed 29

I know that GridSearchCV can take multiple inputs as their scorer, but I will have to choose a metric to optimise using refit.

Like so:

    grid_search = GridSearchCV(
    estimator=classifier,
    param_grid=parameters,
    scoring=['accuracy', 'f1', 'precision', 'recall'],
    refit="accuracy",  # Or any other value from `scoring` list)

As I understand, if I put refit = accuracy here, the grid search will optimise the accuracy, meaning it will find the set of hyperparameters that will have the highest accuracy and I will be able to get the precision, recall and f1 score from cv_result

But what I want to do now is to optimise both accuracy and F1 score as my use case needs both metrics, is there a way to achieve this? I scoured the Internet for the solution but to no avail.

gridsearchcv grid-search machine-learning python
1 Answers

optimize both accuracy and F1 score

This isn't a well-defined optimization problem: what happens if one model has accuracy=0.9 and F1=0.7 and another has accuracy=0.8 and F1=0.85? As such, there can't be a builtin way to just specify "accuracy, F1".

Once you have defined your custom objective more clearly, you can pass a callable for refit; the documentation says this in that case:

Where there are considerations other than maximum score in choosing a best estimator, refit can be set to a function which returns the selected best_index_ given cv_results_. In that case, the best_estimator_ and best_params_ will be set according to the returned best_index_ while the best_score_ attribute will not be available.

So, for example,

def my_refit_criteria(cv_results_):
    return np.argmax(
        cv_results_['mean_test_accuracy']
        + cv_results_['mean_test_f1']
    )
Ben Reiniger0
Related