Keras - Add attention mechanism to an LSTM model

user_1115237 0
Viewed 16084

With the following code:

model = Sequential()

num_features = data.shape[2]
num_samples = data.shape[1]

model.add(
    LSTM(16, batch_input_shape=(None, num_samples, num_features), return_sequences=True, activation='tanh'))
model.add(PReLU())
model.add(Dropout(0.5))
model.add(LSTM(8, return_sequences=True, activation='tanh'))
model.add(Dropout(0.1))
model.add(PReLU())
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))

I'm trying to understand how can I add an attention mechanism before the first LSTM layer. I've found the following GitHub: keras-attention-mechanism by Philippe Rémy but couldn't figure out how exactly to use it with my code.

I would like to visualize the attention mechanism and see what are the features that the model focus on.

Any help would be appreciated, especially a code modification. Thanks :)

attention-model lstm keras machine-learning python
2 Answers
user_31920210

There are at least half a dozen major flavours of attention, most of them are minor variations over the first Attention model that came out - Bahdanau et al in 2014. Each of this flavour can be implemented in multiple ways, so this can be confusing to someone who wants to add a simple attention layer to her/his model. Looking at your model, I would recommend adding an attention layer after your second LSTM layer. This can be a custom attention layer based on Bahdanau.

An implementation is shared here: Create an LSTM layer with Attention in Keras for multi-label text classification neural network

You could then use the 'context' returned by this layer to(better) predict whatever you want to predict. So basically your subsequent layer (the Dense sigmoid one) would use this context to predict more accurately.

The attention weights too are returned by the above layer. These can be routed to a simple display.

For more specific details, please refer https://towardsdatascience.com/create-your-own-custom-attention-layer-understand-all-flavours-2201b5e8be9e

user_146421800
Related