Overfitting in LSTM even after using regularizers

user_7934206 0
Viewed 6607

I am having a time series prediction problem and building an LSTM like below :

def create_model():
    model = Sequential()
    model.add(LSTM(50,kernel_regularizer=l2(0.01), recurrent_regularizer=l2(0.01), bias_regularizer=l2(0.01), input_shape=(train_X.shape[1], train_X.shape[2])))
    model.add(Dropout(0.591))
    model.add(Dense(1))
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

When I train the model on 5 splits like below :

tss = TimeSeriesSplit(n_splits = 5)
X = data.drop(labels=['target_prediction'], axis=1)
y = data['target_prediction'] 
for train_index, test_index in tss.split(X):
   train_X, test_X = X.iloc[train_index, :].values, X.iloc[test_index,:].values
   train_y, test_y = y.iloc[train_index].values, y.iloc[test_index].values
   model=create_model()
   history = model.fit(train_X, train_y, epochs=10, batch_size=64,validation_data=(test_X, test_y), verbose=0, shuffle=False)

I get an overfitting problem. The graph of loss is attached enter image description here

I am not sure why there is overfitting when I use regularizers in my Keras model. Any help is appreciated .

EDIT: Tried the architectures 

def create_model():
    model = Sequential()
    model.add(LSTM(20, input_shape=(train_X.shape[1], train_X.shape[2])))
    model.add(Dense(1))
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model


def create_model(x,y):
    # define LSTM
    model = Sequential()
    model.add(Bidirectional(LSTM(20, return_sequences=True), input_shape=(x,y)))
    model.add(TimeDistributed(Dense(1, activation='sigmoid')))
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

but still it is overfitting.

recurrent-neural-network lstm keras deep-learning python
1 Answers

First of all remove all your regularizers and dropout. You are literally spamming with all the tricks out there and 0.5 dropout is too high.

Reduce the number of units in your LSTM. Start from there. Reach a point where your model stops overfitting.

Then, add dropout if required.

After that, the next step is to add the tf.keras.Bidirectional. If still, you are not satfisfied then, increase number of layers. Remember to keep return_sequences True for every LSTM layer except the last one.

It is seldom I come across networks using layer regularization despite the availability because dropout and layer regularization have a same effect and people usually go with dropout (at maximum, I have seen 0.3 being used).

user_93539090
Related