What is difference between "train_test_split(shuffle=False)" and "TimeSeriesSplit"

user_17601122 0
Viewed 477

When I tried two ways of splitting my time series data, prediction result with test dataset was different. When I checked my data, method1 and 2 shows same result of splitting data. But, the prediction result by test dataset were different. So, what is difference between using train_test_split(shuffle=False) and TimeSeriesSplit?

Here are what I tried: Dataset for split

X = df_5T.drop('demand', axis=1)
y = df_5T.demand

  1. train_test_split

    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.167, shuffle=False, random_state=0)

  2. TimeSeriesSplit

    from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit()
for train_index, val_index in tscv.split(X):
  print("TRAIN:", train_index, "TEST:", val_index)
  X_train, X_val = X.iloc[train_index], X.iloc[val_index]
  y_train, y_val = y.iloc[train_index], y.iloc[val_index]

##Result##
TRAIN: [   0    1    2 ... 1485 1486 1487] TEST: [1488 1489 1490 ... 2973 2974 2975]
TRAIN: [   0    1    2 ... 2973 2974 2975] TEST: [2976 2977 2978 ... 4461 4462 4463]
TRAIN: [   0    1    2 ... 4461 4462 4463] TEST: [4464 4465 4466 ... 5949 5950 5951]
TRAIN: [   0    1    2 ... 5949 5950 5951] TEST: [5952 5953 5954 ... 7437 7438 7439]
TRAIN: [   0    1    2 ... 7437 7438 7439] TEST: [7440 7441 7442 ... 8925 8926 8927]

When I checked data after timeseriessplit, X_train, X_val followed last fold.

X_train.info()
####
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 7440 entries, 2020-01-01 00:00:00+09:00 to 2020-01-26 19:55:00+09:00
####

X_train.info()
####
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 1488 entries, 2020-01-26 20:00:00+09:00 to 2020-01-31 23:55:00+09:00
####

Test data

X_test = data02_5T.drop('demand', axis=1)
y_test = data02_5T.demand

model fit and predict code

model.fit(X_train, y_train)
y_pred = model.predict(X_val)
y_test = model.predict(X_test)
print(rmsle(y_val, y_pred))
print(rmsle(y_test, y_test))

Method1 result

val: 10.659636522389675
test: 136.65172778040608

Method2 result

val: 12.655831132167329
test: 18.771364489679307
scikit-learn pandas python
1 Answers

For time-series related machine learning, you have to use the timeseriesplit() method. Otherwise, it is data leaking. You will get a very high score in lab environment but fail in the real-world stage. Comparing method 1 and method 2 has shown it clearly:

Method1 result val: 10.659636522389675(high score as data leaking) test: 136.65172778040608(failed score as your model is not generalized)

Method2 result val: 12.655831132167329 (low compare the method1 as not data leaking) test: 18.771364489679307( reasonable score, less than validation but good in the real world)

Why method 1 failed and it is data leaking?

Here is the answer. train_test_split() do not design for time series data. it just randomly split data.

Let's say, you want to train data and predict the future. The train data has 5 days data in Jan. enter image description here train_test_split() may use Jan 1st, Jan 2st, Jan 3rd, Jan fifth as training data, to predict Jan fourth. In the real world, Jan Forth is strongly related to Jan 1,2,3,5. That' lead to data leaks.

In reality work, you will only predict the future, instead of the past. So, train_test_split() showed a good validation score but failed in real data when predicting the future.

TimeSeriesSplit() on the roll. It splits data-based accumulated. it makes sure you train/predict the future instead of the past. In this way, the model trained could be useful to predict future data. enter image description here

Here is the detailed documentation regarding TimeSeriesSplit().

If you like my answer, pls vote for it.

Best Regards,

WangYong

user_88671500
Related