CatBoost: Are we overfitting?

user_17204450 0
Viewed 377

Our team is currently using CatBoost to develop credit scoring models, and our current process is to...

  1. Sort the data chronologically for out-of-time sampling, and split it into train, valid, and test sets
  2. Perform feature engineering
  3. Perform feature selection and hyperparameter tuning (mainly learning rate) on train, using valid as an eval set for early stopping
  4. Perform hyperparameter tuning on the combination of train and valid, using test as an eval set for early stopping
  5. Evaluate the results of Step #4 using standard metics (RMSE, ROC AUC, etc.)

However, I am concerned that we may be overfitting to the test set in Step #4.

In Step #4, should we just be refitting the model on train and valid without tuning (i.e., using the selected features and hyperparameters from Step #3)?

The motivation for having a Step #4 at all is to train the models on more recent data due to our out-of-time sampling scheme.

overfitting-underfitting catboost xgboost hyperparameters python
1 Answers

Step #4 falls outside of the best practices for machine learning.

When you create the test set, you need to set it aside and only use it at the end to evaluate how successful your model(s) are at making predictions. Do not use the test set to inform hyperparameter tuning! If you do, you will overfit your data.

Try using cross-validation instead:

enter image description here

user_119406660
Related