How to get reproducible results from XGBoostRegressor? random_state has no effect

user_9116982 0
Viewed 1917

I realized that, contrary to scikit learn, setting a fixed value for random_state does not guarantee that the model will output the same results everytime.

Hence I'm not able to get reproducible results from XGBoostRegressor, even by setting seed, random_state, colsample_bytree and subsample.

Is this a bug? Is this somewhat by design? If so, why?

If you have a solution or a workaround that always works, please share.

Here's the code: 

model = XGBRegressor(n_estimators=1000, learning_rate=0.05,
                     subsample=0.8, colsample_bytree= 0.8, seed=42)

model.fit(X_train_trf,y_train,
        early_stopping_rounds=5,
        eval_set=[(X_train_trf, y_train), (X_valid_trf, y_valid)],
        verbose=False)
preds = model.predict(X_valid_trf)
xgboost machine-learning python-3.x
2 Answers

For reference, the issue was not so much with XGBoost but with the data splitting.

I was splitting the data with train_test_split without setting the random_state, which caused some randomness.

Fixed as below:

X_train_full, X_valid_full, y_train, y_valid = train_test_split(X_full, y,
  train_size=0.8, test_size = 0.2, random_state=1)
user_91169820

In answer to your question, slight differences could be explained by non-determinism in floating point summation order and multi-threading but overall gradient boosting doesn't really lend itself to building models reproducibly. You can control a lot of the randomness by tweaking the parameters (e.g. setting random_state, seed, input features, model params, and the suggestions I made in the comments above), but overall you expect some small variations when building models due to how the algorithm works; especially if the signal in the data isn't strong.

user_129573400
Related