LSTM autoencoder noisy reconstruction

user_12694824 0
Viewed 201

I am trying to build an LSTM autoencoder for the compression of time series (currently only one dimensional, but could also be for multiple dimensions).

A little bit of context first:

I am developing the model using DeepNote and according to the terminal the installed TensorFlow version is 2.4.1 with Keras version 2.4.3 (Linux)

For training data I use roughly 400.000 data points of specific sensors from the Intel Berkeley Research Lab Dataset as training data and another 40.000 as test data. All data points are temperatures collected by the sensors. The data is split into small sequences (usually 20 data points per sequence), thus the resulting shape of the reshaped data is (num_examples, sequence_length, dims) which could look like (19456, 20, 1). The model will make its predictions on one of those 20-length sequences at a time.

This is the code for the simplest form of the model

n_dims = data.shape[2]
sample_size = data.shape[1]

inputs = Input(shape=(data.shape[1],n_dims), batch_size=batch_size)

# encoder
x = LSTM(n_dims, activation='relu', return_sequences=False, stateful=True, kernel_regularizer=regularizers.l2(regularization))(inputs)
    
# decoder
# Because sequences are not returned by encoder for (better) compression, have to be spread
# for decoder
x = RepeatVector(sample_size)(x)

decoding = LSTM(n_dims, activation='relu', return_sequences=True, stateful=True, kernel_regularizer=regularizers.l2(regularization))(x)


model = Model(inputs=inputs, outputs=decoding)
model.compile(optimizer=optimizer, loss=loss)

Model summary:

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(1, 20, 1)]              0         
_________________________________________________________________
lstm (LSTM)                  (1, 1)                    12        
_________________________________________________________________
repeat_vector (RepeatVector) (1, 20, 1)                0         
_________________________________________________________________
lstm_1 (LSTM)                (1, 20, 1)                12        
=================================================================
Total params: 24
Trainable params: 24
Non-trainable params: 0

This is the code for training the model:

model.fit(
    x_train, x_train,    # learns an identity function, thus x_train is label and input at the same time
    epochs=num_epochs, batch_size=batch_size,
    shuffle=False,
    validation_data=(x_test, x_test),
    verbose=verbosity
)

My problem:

The model is more or less able to reconstruct the trend that the data follows, however if one zooms in, there is this weird periodic behaviour. Following are images of what the output of the model looks like. When displaying the outputs I concatenate each of the sequences together such that I get the original shape of the data back. The first picture shows reconstruction zoomed in on 5000 data points and the second one on 100 data points

What's puzzling is that the length of the period is always equal to the sequence length (e.g 20 as in the above code). If I would change it to 40, those bumps would occur every 50 data points. My theory is that this is due to the LSTM cells resetting their state after each training example and therefore not being able to 'carry over' any state or context into a new one. Reading through the internet I set the stateful parameter to true because that would prevent the model from resetting the state as I understood it. Also for training I set shuffle=False such that the sequences are in the right order.

autoencoder lstm keras tensorflow python
1 Answers

I think I fixed the issue - or rather found a workaround.

The problem was indeed the stateful-related component. The LSTM layer will by default reset its state after it has processed one batch of data. Resetting in this case means that the all of its states are set to only zeros again. You can also see in the reconstructed sequences that at the beginning of each batch the reconstructed value was too small.

Setting the stateful parameter to true will tell the layer to maintain its state. Yet, if the batch size is greater than 1, multiple parts of the sequence will be computed in parallel and therefore not be able to share information about the state. Thus by setting the batch size to 1, the issue does not appear again, however training now takes longer.

user_126948240
Related