LSTM-based architecture for EEG signal Classification based-on "Channel LSTM"

Question

I have a multi-class Classification issue that I use of keras & tensorflow in python 3.6. I have a good implementation for my classification with high accuracy based on "stacked LSTM layers (a)" that mention in this article: Deep Learning Human Mind for Automated Visual Classification.

some thing like this:

model.add(LSTM(256,input_shape=(32, 15360), return_sequences=True))
model.add(LSTM(128), return_sequences=True)
model.add(LSTM(64), return_sequences=False)

model.add(Dense(6, activation='softmax'))

let 32 is # of EEG channels and 15360 is the length of the signal by 160 Hz in 96 second recording

I want to implement "the Channel LSTM and Common LSTM (b)" strategy that mention in above article but I don't know how should I make my model by this new strategy.

Please help me. Thx

user_13509540 · Answer

First, you have a problem in your implementation of encoder using Common LSTM, the LSTM layer of keras take inputs with shape (batch, timesteps, channel) by default, so if you set your input_shape=(32, 15360) then the model will read as timesteps=32 and channel=15360 which is opposite of what you intend to.

Because first layer of encoder using Common LSTM described as:

At each time step t, the first layer takes the input s(·, t)(in this sense, “common” means that all EEG channels are initially fed8 into the same LSTM layer)

So the correct implementation of encoder using Common LSTM would be:

import tensorflow as tf
from tensorflow.keras import layers, models

timesteps = 15360
channels_num = 32

model = models.Sequential()
model.add(layers.LSTM(256,input_shape=(timesteps, channels_num), return_sequences=True))
model.add(layers.LSTM(128, return_sequences=True))
model.add(layers.LSTM(64, return_sequences=False))
model.add(layers.Dense(6, activation='softmax'))

model.summary()

Which outputs (PS: you could summarize your original implementation and you will see the Total params is way bigger):

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm (LSTM)                  (None, 15360, 256)        295936
_________________________________________________________________
lstm_1 (LSTM)                (None, 15360, 128)        197120
_________________________________________________________________
lstm_2 (LSTM)                (None, 64)                49408
_________________________________________________________________
dense (Dense)                (None, 6)                 390
=================================================================
Total params: 542,854
Trainable params: 542,854
Non-trainable params: 0
_________________________________________________________________

Second, because encoder using Channel LSTM and Common LSTM described as:

The first encoding layer consists of several LSTMs, each connected to only one input channel: for example, the first LSTM processes input datas(1,·), the second LSTM processess(2,·), and so on. In this way, the output of each “channel LSTM”is a summary of a single channel’s data. The second encoding layer then performs inter-channel analysis, by receiving as input the concatenated output vectors of all channel LSTMs. As above, the output of the deepest LSTM at the last time step is used as the encoder’s output vector.

Since each LSTM in the first layer only deal with one channel, so we need the number of LSTM equal to number of channel in the first layer, following code show you how to build one encoder using Channel LSTM and Common LSTM:

import tensorflow as tf
from tensorflow.keras import layers, models

timesteps = 15360
channels_num = 32

first_layer_inputs = []
second_layer_inputs = []
for i in range(channels_num):
    l_input = layers.Input(shape=(timesteps, 1))
    first_layer_inputs.append(l_input)
    l_output = layers.LSTM(1, return_sequences=True)(l_input)
    second_layer_inputs.append(l_output)

x = layers.Concatenate()(second_layer_inputs)
x = layers.LSTM(128, return_sequences=True)(x)
x = layers.LSTM(64, return_sequences=False)(x)
outputs = layers.Dense(6, activation='softmax')(x)

model = models.Model(inputs=first_layer_inputs, outputs=outputs)

model.summary()

Outputs:

Model: "functional_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_2 (InputLayer)            [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_3 (InputLayer)            [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_4 (InputLayer)            [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_5 (InputLayer)            [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_6 (InputLayer)            [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_7 (InputLayer)            [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_8 (InputLayer)            [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_9 (InputLayer)            [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_10 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_11 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_12 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_13 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_14 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_15 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_16 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_17 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_18 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_19 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_20 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_21 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_22 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_23 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_24 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_25 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_26 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_27 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_28 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_29 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_30 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_31 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
input_32 (InputLayer)           [(None, 15360, 1)]   0
__________________________________________________________________________________________________
lstm (LSTM)                     (None, 15360, 1)     12          input_1[0][0]
__________________________________________________________________________________________________
lstm_1 (LSTM)                   (None, 15360, 1)     12          input_2[0][0]
__________________________________________________________________________________________________
lstm_2 (LSTM)                   (None, 15360, 1)     12          input_3[0][0]
__________________________________________________________________________________________________
lstm_3 (LSTM)                   (None, 15360, 1)     12          input_4[0][0]
__________________________________________________________________________________________________
lstm_4 (LSTM)                   (None, 15360, 1)     12          input_5[0][0]
__________________________________________________________________________________________________
lstm_5 (LSTM)                   (None, 15360, 1)     12          input_6[0][0]
__________________________________________________________________________________________________
lstm_6 (LSTM)                   (None, 15360, 1)     12          input_7[0][0]
__________________________________________________________________________________________________
lstm_7 (LSTM)                   (None, 15360, 1)     12          input_8[0][0]
__________________________________________________________________________________________________
lstm_8 (LSTM)                   (None, 15360, 1)     12          input_9[0][0]
__________________________________________________________________________________________________
lstm_9 (LSTM)                   (None, 15360, 1)     12          input_10[0][0]
__________________________________________________________________________________________________
lstm_10 (LSTM)                  (None, 15360, 1)     12          input_11[0][0]
__________________________________________________________________________________________________
lstm_11 (LSTM)                  (None, 15360, 1)     12          input_12[0][0]
__________________________________________________________________________________________________
lstm_12 (LSTM)                  (None, 15360, 1)     12          input_13[0][0]
__________________________________________________________________________________________________
lstm_13 (LSTM)                  (None, 15360, 1)     12          input_14[0][0]
__________________________________________________________________________________________________
lstm_14 (LSTM)                  (None, 15360, 1)     12          input_15[0][0]
__________________________________________________________________________________________________
lstm_15 (LSTM)                  (None, 15360, 1)     12          input_16[0][0]
__________________________________________________________________________________________________
lstm_16 (LSTM)                  (None, 15360, 1)     12          input_17[0][0]
__________________________________________________________________________________________________
lstm_17 (LSTM)                  (None, 15360, 1)     12          input_18[0][0]
__________________________________________________________________________________________________
lstm_18 (LSTM)                  (None, 15360, 1)     12          input_19[0][0]
__________________________________________________________________________________________________
lstm_19 (LSTM)                  (None, 15360, 1)     12          input_20[0][0]
__________________________________________________________________________________________________
lstm_20 (LSTM)                  (None, 15360, 1)     12          input_21[0][0]
__________________________________________________________________________________________________
lstm_21 (LSTM)                  (None, 15360, 1)     12          input_22[0][0]
__________________________________________________________________________________________________
lstm_22 (LSTM)                  (None, 15360, 1)     12          input_23[0][0]
__________________________________________________________________________________________________
lstm_23 (LSTM)                  (None, 15360, 1)     12          input_24[0][0]
__________________________________________________________________________________________________
lstm_24 (LSTM)                  (None, 15360, 1)     12          input_25[0][0]
__________________________________________________________________________________________________
lstm_25 (LSTM)                  (None, 15360, 1)     12          input_26[0][0]
__________________________________________________________________________________________________
lstm_26 (LSTM)                  (None, 15360, 1)     12          input_27[0][0]
__________________________________________________________________________________________________
lstm_27 (LSTM)                  (None, 15360, 1)     12          input_28[0][0]
__________________________________________________________________________________________________
lstm_28 (LSTM)                  (None, 15360, 1)     12          input_29[0][0]
__________________________________________________________________________________________________
lstm_29 (LSTM)                  (None, 15360, 1)     12          input_30[0][0]
__________________________________________________________________________________________________
lstm_30 (LSTM)                  (None, 15360, 1)     12          input_31[0][0]
__________________________________________________________________________________________________
lstm_31 (LSTM)                  (None, 15360, 1)     12          input_32[0][0]
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, 15360, 32)    0           lstm[0][0]
                                                                 lstm_1[0][0]
                                                                 lstm_2[0][0]
                                                                 lstm_3[0][0]
                                                                 lstm_4[0][0]
                                                                 lstm_5[0][0]
                                                                 lstm_6[0][0]
                                                                 lstm_7[0][0]
                                                                 lstm_8[0][0]
                                                                 lstm_9[0][0]
                                                                 lstm_10[0][0]
                                                                 lstm_11[0][0]
                                                                 lstm_12[0][0]
                                                                 lstm_13[0][0]
                                                                 lstm_14[0][0]
                                                                 lstm_15[0][0]
                                                                 lstm_16[0][0]
                                                                 lstm_17[0][0]
                                                                 lstm_18[0][0]
                                                                 lstm_19[0][0]
                                                                 lstm_20[0][0]
                                                                 lstm_21[0][0]
                                                                 lstm_22[0][0]
                                                                 lstm_23[0][0]
                                                                 lstm_24[0][0]
                                                                 lstm_25[0][0]
                                                                 lstm_26[0][0]
                                                                 lstm_27[0][0]
                                                                 lstm_28[0][0]
                                                                 lstm_29[0][0]
                                                                 lstm_30[0][0]
                                                                 lstm_31[0][0]
__________________________________________________________________________________________________
lstm_32 (LSTM)                  (None, 15360, 128)   82432       concatenate[0][0]
__________________________________________________________________________________________________
lstm_33 (LSTM)                  (None, 64)           49408       lstm_32[0][0]
__________________________________________________________________________________________________
dense (Dense)                   (None, 6)            390         lstm_33[0][0]
==================================================================================================
Total params: 132,614
Trainable params: 132,614
Non-trainable params: 0
__________________________________________________________________________________________________

Now because the model expect inputs with shape (channel, batch, timesteps, 1), so we have to reorder the axis of dataset before feed into model, Following example code show you how to reorder the axis from (batch, timesteps, channel) to (channel, batch, timesteps, 1):

import numpy as np

batch_size = 64
timesteps = 15360
channels_num = 32

x = np.random.rand(batch_size, timesteps, channels_num)
print(x.shape)
x = np.moveaxis(x, -1, 0)[..., np.newaxis]
print(x.shape)
x = [i for i in x]
print(x[0].shape)

Outputs:

(64, 15360, 32)
(32, 64, 15360, 1)
(64, 15360, 1)

LSTM-based architecture for EEG signal Classification based-on "Channel LSTM"

1 Answers