What is the meaning of min_sum_hessian_in_leaf in lightgbm (see http://lightgbm.readthedocs.io/en/latest/Parameters.html)? I know that the hessian is a matrix of second order derivatives but I don't understand what that means in the context of lightgbm (or gradient boosting in general). And how does lightgbm condense that matrix down into a single value?
2 Answers
Simple answer:
The cover (aka min_sum_hessian_in_leaf ) is just a parameter for taking account the number of observations in a leaf of our split (actually is more than that). When doing a split in the m=1,..,M estimators; If the sum of the hessian in a leaf is lower than min_sum_hessian_in_leaf the tree will stop growing. For example, in the simplest case, when we are talking about regression and the loss function is the typical (1/2)*sum(y_i-ypred_i )^2 (no regularization) then sum_hessian_in_leaf is equal to the number of observations in the leaf. So let's say min_sum_hessian_in_leaf = 3 ; therefore we will require at least 3 observations in each leaf for considering a new split. So if min_sum_hessian_in_leaf = 0 , the m-tree (remember gbm are m=1,..,M trees/estimators) will grow freely and of course, trees that grow without constraints tend to overfit. On the otherside is min_sum_hessian_in_leaf is high enough, the m=1,..,M will have low complexity (like stumps/ short trees).
More complex (my interpretation):
At this point you might ask yourself why we are talking about the hessian. The answer is quite complex, but one way of resolving the GBM algorithm (see the algorithm here https://en.wikipedia.org/wiki/Gradient_boosting ) iS instead of minimizing the actual Loss function is minimizing an approximation of a loss function. Which is taking a second order polynomial around 0 (this will allow us to solve the problem by newton-raphson algorithm). So the loss function to minimize now depends on the first derivative (which can also be thought as the jacobian) and the second derivative (which can also be thought as the hessian). At this point, we know that in a context of decission trees, when we have a new split that have a high hessian, this means the split reduces the loss very good (because of the approximation, actually we are trying to reduce the gradient of the loss in the context of gbm). So when min_sum_hessian_in_leaf > sum_hessian_in_leaf this means our split is not good enough and therefore we cannot make the split in our tree. On the other side if min_sum_hessian_in_leaf <= sum_hessian_in_leaf , this means our split is good enough at reducing the loss (actually the gradient of the loss) and therefore we can keep the tree growing. The formula for sum_hessian_in_leaf varies depending of the loss function. So you bet set a different min_sum_hessian_in_leaf depending on the loss function you are trying to minimize (in a regression problem - no regularization the sum_hessian_in_leaf = number of observations in the leaf , as I stated before). ´
References:
The youtuber joshstarmer explains these kind of details on his playlists of xgboost/gbms. The elements of statistical learning (tibshirani et al) has a chapter about gbms but not about lightgbm. But this helps a lot. XGBOOST paper: https://arxiv.org/pdf/1603.02754.pdf (you can also read lgbm paper but this is more helpful for me since the algorithms are really alike except for the part of EFB and GOOS + other details).
Related