LightGBM depth not increasing from zero

user_19409676 0
Viewed 32

I am trying to get familiar with LightGBM. Suppose data is generated following the data generating process:

x = [1, 2, 3, 4, 5]

y = 0.5 * x

And we split the data keeping the first 4 observations in the training set and the remaining 1 observation in the test set.

If I make a regression tree with a depth of 1 (and 2 leaves) I would expect it to split the training data in half resulting in leaves of

leaf-1: x < 2.5

leaf-2: x >= 2.5

However when I try this using LightGBM it is returning a tree without any leaves. Any help with why the parameters I set is not resulting in the expected outcome?

Code for reproducibility:

import pandas as pd
import lightgbm as lgb

df_train = pd.DataFrame({"x": [1, 2, 3, 4], "y": [0.5, 1, 1.5, 2]}).to_numpy()
df_test  = pd.DataFrame({"x": [5], "y": [2.5]}).to_numpy()

lgb_df_train = lgb.Dataset(df_train[:,0].reshape(-1,1), label = df_train[:,1])
lgb_df_test = lgb.Dataset(df_test[:,0].reshape(-1,1), label = df_test[:,1])

lgb_params = {
        "objective": "regression",
        "metric": "",
        "verbose": -1,
        "early_stopping_round" : 5,
        "n_estimators": 1,
        "max_depth": 2, 
        "min_data_in_leaf": 2,
}

tree = lgb.train(params = lgb_params, train_set = lgb_df_train, valid_sets = [lgb_df_test])

tree_df = tree.trees_to_dataframe()

pd.DataFrame(tree_df)
tree_index node_depth node_index left_child right_child parent_index split_feature split_gain threshold decision_type missing_direction missing_type value weight count
0 1 0-L0 1.25
lightgbm python
1 Answers

LightGBM discretizes continuous features into histograms. To prevent overfitting, it provides a parameter min_data_in_bin, which controls how small the histogram bins are.

That parameter defaults to 3, which makes all splits impossible for a dataset as small as the one in your example which only has 4 rows.

Set min_data_in_bin=1 and min_data_in_leaf=1 to fit a LightGBM model to such a small dataset.

import pandas as pd
import lightgbm as lgb

df_train = pd.DataFrame(
    {"x": [1, 2, 3, 4], "y": [0.5, 1, 1.5, 2]}
).to_numpy()

dtrain = lgb.Dataset(
    data=df_train[:,0].reshape(-1,1),
    label=df_train[:,1],
    params={
        "min_data_in_bin": 1
    }
)

lgb_params = {
    "objective": "regression",
    "verbose": -1,
    "n_estimators": 1,
    "max_depth": 2, 
    "min_data_in_leaf": 2,
}

tree = lgb.train(
    params = lgb_params,
    train_set = dtrain
)

tree.trees_to_dataframe()
tree_index node_depth node_index left_child right_child parent_index split_feature split_gain threshold decision_type missing_direction missing_type value weight count
0 0 1 0-S0 0-L0 0-L1 Column_0 1 2.5 <= left None 1.25 0 4
1 0 2 0-L0 0-S0 nan nan 1.2 2 2
2 0 2 0-L1 0-S0 nan nan 1.3 2 2
user_39866770
Related