Hyperparameter Grid Search with XGBoost - Scoring function vs Evaluation Metric

user_7762646 0
Viewed 2735

Dealing with an imbalance dataset problem (7% vs 93%), I want to find out the best structure of my xgboost model using grid search cross-validation. Note: I am using stratified k-fold cross-validation to make sure each fold has the correct proportion of the minority class.

My situation is the following, I noticed GridSearchCV has a parameter called 'scoring' to which I can pass even more than one sklearn.metrics as shown here. However, xgboost has also a parameter called 'eval_metric' and I am a bit confused between the two. Could you explain me the difference if any? Where is the correct place to specify it?

I will attach a little bit of code to illustrate it: 

params_grid = {
'n_estimators': [100, 300],
'learning_rate': [0.01, 0.1, 0.3],
'colsample_bytree': [0.3, 0.5],
}

params_fixed = {
    'objective':'binary:logistic',
    'silent':1,
    'eval_metric':'auc'
}


n_folds = 5
skf = StratifiedKFold(n_splits=n_folds,random_state=seed,shuffle=True)

# create xgboost classifier
xgb = XGBClassifier(**params_fixed, seed=seed)

grid_search = GridSearchCV(estimator=xgb, param_grid=params_grid, 
                               cv=skf.split(X_train, y_train), scoring='balanced_accuracy')

In addition, which function would you then recommend to use in my case? It can be from skearn.metrics or even a custom-one but I do not know yet how to write it. Note, my problem is a trade-off problem between recall and precision but recall is to me the most important since I would like to detect the minority class in 99% cases.

xgboost data-science scikit-learn machine-learning python
1 Answers

Turning my comment into an answer, there is no bypass whatsoever and everything still works, but it just doesn't make sense. Every algorithm maximizes the metric you tell it to, so in your example xgboost will build trees to maximize the auc, and the grid search will find the hyper-parameters that maximize the accuracy. And it clearly makes no sense.

So you should set both metrics to the same, be it AUC, recall or whatever you see fit for your problem; for imbalanced datasets AUC is a good choice, or you could go for a F-score that is more balanced towards recall.

user_5217760
Related