The predictions from StackingRegressor (Sklearn) are not reproducible

user_11003528 0
Viewed 137

I am working on training a regression model using StackingRegressor and I found out that the prediction from this model is not consistent while I am using the same random_state.

Here is my code:

random_seed = 42
mdl_lgbm = lightgbm.LGBMRegressor(colsample_bytree=0.6,
                                      learning_rate=0.05,
                                      max_depth=6,
                                      min_child_samples=227,
                                      min_child_weight=10,
                                      n_estimators=1800,
                                      num_leaves=45,
                                      reg_alpha=0,
                                      reg_lambda=1,
                                      subsample=0.6,
                                      n_jobs=-1,
                                      random_state=random_seed)



mdl_xgb = xgb.XGBRegressor(subsample=0.5,
                                n_estimators=900,
                                min_child_weight=8,
                                max_depth=6,
                                learning_rate=0.03,
                                colsample_bytree=0.8,
                                n_jobs=-1,
                                reg_alpha=2,
                                reg_lambda=50,
                                objective='reg:squarederror',
                                random_state=random_seed)


mdl_rf = RandomForestRegressor(bootstrap=True,
                                   max_depth=110,
                                   max_features='auto',
                                   min_samples_leaf=5,
                                   min_samples_split=5,
                                   n_estimators=1430,
                                   n_jobs=-1,
                                   random_state=random_seed)


# Base models
base_mdl_names = {
    'XGB': mdl_xgb,
    'LGBM': mdl_lgbm,
    'RF': mdl_rf,
}

        
        
final_estimator = xgb.XGBRegressor(subsample=0.3,
                                   n_estimators=1200,
                                   min_child_weight=2,
                                   max_depth=5,
                                   learning_rate=0.06,
                                   colsample_bytree=0.8,
                                   n_jobs=-1,
                                   reg_alpha=1,
                                   reg_lambda=0.1,
                                   objective='reg:squarederror',
                                   random_state=random_seed)

base_estimators = list()
for name, mdl in base_mdl_names.items():
    base_estimators.append((name, mdl))

stacked_mdl = StackingRegressor(estimators=base_estimators,
                                final_estimator=final_estimator,
                                cv=5,
                                passthrough=True)

stacked_mdl.fit(X_train, y_train)

Please note that I am not changing X_train. When I use the trained model for the prediction, the results are not reproducable. I mean that if I re-train the model, the results will be different while every input is the same. Any clue why this happening?

scikit-learn regression machine-learning python
1 Answers

StackingRegressor use cv (cross validation). Therefore you also have to set its random_state in order to have exactly the same cross validation split at each run.

You should do as follow:

from sklearn.model_selection import KFold
kfold = KFold(n_splits=5, random_state=random_seed, shuffle=True)
stacked_mdl = StackingRegressor(estimators=base_estimators,
                                final_estimator=final_estimator,
                                cv=kfold,
                                passthrough=True)
user_45746330
Related