"reg_alpha" parameter in XGBoost regressor. Is it bad to use high values?

user_11353834 0
Viewed 4808

I'm performing hyperparameter tuning with grid search and I realized that I was getting overfitting... I tried a lot of ways to reduce it, changing the "gamma", "subsample", "max_depth" parameters to reduce it, but I was still overfitting...

Then, I increased the "reg_alpha" parameters value to > 30....and them my model reduced overfitting drastically. I know that this parameter refers to L1 regularization term on weights, and maybe that's why solved my problem.

I just want to know if it has any problem using high values for reg_alpha like this?

I would appreciate your help :D

xgboost python
1 Answers

reg_alpha penalizes the features which increase cost function. Meaning it finds the features that doesn't increase accuracy. But this makes prediction line smoother.

On some problems I also increase reg_alpha > 30 because it reduces both overfitting and test error. But if it is a regression problem it's prediction will be close to mean on test set and it will maybe not catch anomalies good.

So i may say you can increase it as long as your test accuracy doesn't begin to fall down.

Lastly when increasing reg_alpha , keeping max_depth small might be a good practise.

user_123472080
Related