XGBoost - Correct usage of early_stopping_rounds combined with Purged Cross Validation

Viewed 391

I am working on an XGBoost classifier on time-series data and I would like to combine the early_stopping_rounds option with the Purged Cross Validation (see for reference https://stats.stackexchange.com/questions/443159/what-is-combinatorial-purged-cross-validation-for-time-series-data ). I have the following code:

# Purged Cross Validation
index_train1 = np.arange(0,len(X_train))

def split(a, n):
    k, m = divmod(len(a), n)
    return (a[i * k + min(i, m):(i + 1) * k + min(i + 1, m)] for i in range(n))

k = 6  # this means that the cross-validation will be run over (6*5)/2 = 15 folds

splits = list(split(index_train1, k))

purged_train = []

for m in range(0,len(splits)):
    for n in range(len(splits)-m-2, -1, -1):
        purged_train.append(splits[m])

purged_validation = []

l = 1
while l <= len(splits):
    for j in range(l,len(splits)):
        purged_validation.append(splits[j])
    l+=1
    
purged_df = pd.DataFrame({'train_index':purged_train, 'val_index': purged_validation})

# XGBoost Classifier
params={
    'n_estimators':3000,
    'objective': 'binary:logistic',
    'learning_rate': 0.005,
    'subsample':0.555,
    'colsample_bytree':0.7,
    'min_child_weight':3,
    'max_depth':8,
    'n_jobs' : -1
}

booster = xgboost.XGBClassifier(parameters=params)
cv_score = []


for kfold in range(0,len(purged_df)):
    print('Fold no. ' + str(kfold+1))
    X_train1 = X_train[purged_df['train_index'][kfold].tolist()]
    X_val1 = X_train[purged_df['val_index'][kfold]]
    y_train1 = y_train.iloc[purged_df['train_index'][kfold]]
    y_val1 = y_train.iloc[purged_df['val_index'][kfold]]
    booster = booster.fit(X_train1,y_train1, eval_set=[(X_train1, y_train1), (X_val1, y_val1)], early_stopping_rounds=100, eval_metric=["auc"], verbose = 1000)
    cv_score.append(booster.score(X_val1, y_val1))

I have noticed that by varying the value of early_stopping_rounds, there is no change on the test accuracy, computed as follows:

from sklearn.metrics import accuracy_score
y_pred = booster.predict(X_test, ntree_limit=booster.best_ntree_limit)
y_pred = pd.Series(y_pred, index = y_test.index, name='pred')
print('Accuracy Test: %s %%' % round(accuracy_score(y_test, y_pred)*100,2))

I suspect that there is something going wrong here, with the code probably only taking into consideration the last iteration of the loop, disregarding the whole cross-validated with early_stopping rounds process, does anyone have any idea on how to fix this? Thanks

0 Answers
Related