XG-Boost: predict(...,predleaf = T) - What does the outcome mean?

user_9524424 0
Viewed 213

I'm not sure what the outcome of xgboost -> predict(..., predleaf=T) is.

The docs state that:

When predleaf = TRUE, the output is a matrix object with the number of columns corresponding to the number of trees.

However, what do the numeric values in the columns stand for?

Minimal example:

library(ISLR)
library(xgboost)
auto = ISLR::Auto
auto$name<-NULL
auto$origin<-NULL

dtrain <- xgb.DMatrix(data=as.matrix(auto[,-1]),label=as.matrix(auto[,1]))

param <- list(booster = 'gbtree'
              , objective = 'reg:squarederror'
              , learning_rate = 0.1
              , set.seed = 2020)

xgb <- xgb.train(params = param
                 , data = dtrain
                 , nrounds = 4)

xgb_leaf <- data.frame(predict(xgb, dtrain, predleaf = T))
head(xgb_leaf)

  X1 X2 X3 X4
1  6  9 11 15
2  6 11 11 15
3  6 11 11 15
4  6 11 11 15
5  6 11 11 15
6  6 11 11 13

So for each tree there is a numerical value for each row/observation.

I suppose this is the "number" of the leaf in which a row/observation falls in a particular tree. Is this correct?

xgboost r
1 Answers

Great question. I believe you are correct in your interpretation. My understanding is that the values printed with predleaf = T represent the node corresponding to the leaf. You can visualise this by printing the tree with xgb.model.dt.tree, e.g.

#install.packages("ISLR")
library(ISLR)
library(xgboost)

options(max.print = 10000)

auto = ISLR::Auto
auto$name<-NULL
auto$origin<-NULL

dtrain <- xgb.DMatrix(data=as.matrix(auto[,-1]),label=as.matrix(auto[,1]))

param <- list(booster = 'gbtree'
              , objective = 'reg:squarederror'
              , learning_rate = 0.1)

xgb <- xgb.train(params = param
                 , data = dtrain
                 , nrounds = 4)

xgb_leaf <- data.frame(predict(xgb, dtrain, predleaf = T))
xgb_leaf

xgb_dt_tree <- data.frame(xgb.model.dt.tree(feature_names = NULL, model = xgb))
xgb_dt_tree

The xgb_dt_tree shows the first tree has leaves at nodes 4, 5, 6, 7 & 8. Column X1 in xgb_leaf contains one of these values for each subject. The second tree has nodes 3, 7, 8, 9, 10, 11 & 12, and Column X2 in xgb_leaf only contains these values.

user_129573400
Related