GridSearchCV for TImeSeriesSplit for time series cross validation

Question

I am training a binary classification model on a time series dataset. I need to apply cross-validation to time series data. A few challenges that I am facing:

Calculate statistics on the training set and apply those statistics to the test set, which I have done.
Apply GridSearchCV on TimeSeries data preserving the above statistics to discover hyperparameters that give best test score.

To Calculate Statistics

class DatasetStatistics:
    def __init__(self, x_train, x_test):
        self.x_train = x_train
        self.x_test = x_test
    
    def compute_highest_high_lowest_low(self):
        self.highest_high = self.x_train["High"].max()
        self.lowest_low   = self.x_train["Low"].min()
    
    def compute_highest_lowest_close(self):
        self.highest_close = self.x_train["Close"].max()
        self.lowest_close = self.x_train["Close"].min()
    
    def set_window(self, window):
        self. Window = window

To solve second problem, I am using

tscv = TimeSeriesSplit(n_splits=5)

results = []

for train_idx, test_idx in tscv.split(df):
    df_train = df.iloc[train_idx]
    df_test  = df.iloc[test_idx]
    
    _df = pd.concat([df_train, df_test])

    num_of_train_observations = int(_df.shape[0] * 0.65)

    x_train = _df.loc[:num_of_train_observations, ].copy()
    x_test = _df.loc[num_of_train_observations:, ].copy()
    

    # I am using statistics computed from this class for further transformations and adding new columns in the dataset.
    statistics = DatasetStatistics(x_train, x_test)
    statistics.compute_highest_lowest_close()
    statistics.compute_highest_high_lowest_low()
    statistics.set_window(10)


    train = _df.iloc[train_idx]
    test = _df.iloc[train.shape[0]:]

    X = train[features[:-1]]
    y = train[features[-1]]
    
    param_grid = {'n_estimators': [x for x in range(1, 800, 100)],
                'gamma': [0.1, 0.5, 0.8, 1.6, 3.2],
                'learning_rate': [0.0001, 0.01, 0.2, 0.5],
                'max_depth': [5, 8, 15, 20, 30],
                "tree_method":["gpu_hist"],
                'reg_alpha': [0.1, 0.5, 1.5, 2.4, 3.2, 6.4],
                'reg_lambda': [0.1, 0.5, 1.5, 2.4, 3.2, 6.4]}
    
    model = XGBClassifier()
    

    # I can set slice to None, and perform no cross-validation but still search for the best hyperparameters, under each iteration of above loop.
    cv = [(slice(None), slice(None))]
    search = GridSearchCV(model, param_grid, scoring="accuracy", n_jobs=2, cv=cv, refit=True)

    print("Fitting in 1st Loop.")

    search. Fit(X, y)

In the above code, the problem is there is no validation being performed, and any best model found, may overfit to test data. The solution I am looking for is:

How to use GridSearchCV on TimeSeriesSplit that can also test the model on the test part of the fold, while preserving the transformation applied to each fold?

GridSearchCV for TImeSeriesSplit for time series cross validation

0 Answers