How could we use Bahdanau attention in a stacked LSTM model?

user_8886590 0
Viewed 222

I aim to use attention in a stacked LSTM model, but I don't know how to add AdditiveAttention mechanism of Keras between encoder and decoder layers. Let say, we have an input layer, an encoder, and a decoder, and a dense classification layer, and we aim our decoder to pay attention on all the hidden states of the encoder (h = [h1, ..., hT]) in deriving its outputs. Is there any high-level coding using the Keras whereby I can do? For example,

input_layer = Input(shape=(T, f))
x = input_layer  
x = LSTM(num_neurons1, return_sequences=True)(x)
# Adding attention here, but I don't know how?
x = LSTM(num_neurons2)(x)
output_layer = Dense(1, 'sigmoid')(x)
model = Model(input_layer, output_layer)
...

I think this is wrong to use: x = AdditiveAttention(x, x). Am I right?

attention-model lstm keras
1 Answers

Maybe it is helpful for your issue ?

This is a classification model with LSTM and attention for classification on character-level:

first create a custom layer for attention : class attention(Layer): def init(self,**kwargs): super(attention,self).init(**kwargs)

def build(self,input_shape):
    self.W=self.add_weight(name='attention_weight', shape=(input_shape[-1],1), 
                           initializer='random_normal', trainable=True)
    self.b=self.add_weight(name='attention_bias', shape=(input_shape[1],1), 
                           initializer='zeros', trainable=True)        
    super(attention, self).build(input_shape)

def call(self,x):
    # Alignment scores. Pass them through tanh function
    e = K.tanh(K.dot(x,self.W)+self.b)
    # Remove dimension of size 1
    e = K.squeeze(e, axis=-1)   
    # Compute the weights
    alpha = K.softmax(e)
    # Reshape to tensorFlow format
    alpha = K.expand_dims(alpha, axis=-1)
    # Compute the context vector
    context = x * alpha
    context = K.sum(context, axis=1)
    return context

LEN_CHA = 64 # number of characters LEN_Input = 110 # depend on the longest sentence, padded with zero

def LSTM_model_attention(Labels=3):
model = Sequential()
model.add(Embedding(LEN_CHA, EMBEDDING_DIM, input_length=LEN_INPUT))
model.add(SpatialDropout1D(0.7))
model.add(Bidirectional(LSTM(256, return_sequences=True)))
model.add(attention())
model.add(Dense(Labels, activation='softmax'))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc'])
return model

LSTM_attention = LSTM_model_attention() LSTM_attention.summary()

user_174961130
Related