How to load pre-trained word embeddings in keras and output different classes

user_10640491 0
Viewed 287

I want to compare two LSTMs, one trained on Wikipedia data and one on my own. I have a problem when designing the first model. I defined two word embeddings vocabularies, one that is only built on wikipedia data and one that is also trained on my corpus. I would like to define a first LSTM that has an embedding layer (which loads the pre-trained weights from Wikipedia), but the output should be different than its vocabulary size(10000), in fact I would like to output as many classes as the ones contained into the other vocabulary (50000). This is what I have now:

model = Sequential()
model.add(Embedding(vocab_size_wikipedia, embedding_size, input_length=55, weights=[pretrained_weights_wikipedia])) 
model.add(Bidirectional(LSTM(units=embedding_size)))
model.add(Dense(vocab_size, activation='softmax'))
model.compile(loss='categorical_crossentropy',
          optimizer = RMSprop(lr=0.0005),
          metrics=['accuracy'])

model.fit(np.array(X_train), np.array(y_train), epochs=10, validation_data=(np.array(X_val), np.array(y_val)))

Here my variables and shapes:

shape of pretrained_weights_wikipedia = (10000, 100)
vocab_size = 50000
embedding_size = 100
vocab_size_wikipedia = 10000

X_train.shape() = (1600,55)
y_train.shape() = (1600,50000)
X_train.shape() = (400,55)
X_train.shape() = (400,50000)
X_train.shape() = (200,55)
X_train.shape() = (200,50000) #the labels are padded

Thank you for your help!

lstm word-embedding keras python
1 Answers

You want to build a model in which you want the pretrained weights with different vocabulary size (here 10000) and finetune it for your own corpus with different vocabulary size (here 50000).

Because the problem is symmetric, the mechanism that encodes the first sentence should be reused (weights and all) to encode the second sentence. Here we use a shared layer to encode the inputs.

Click here for more information about shared layers.

input = Input(shape=55)

emb1 = Embedding(vocab_size_wikipedia, embedding_size, weights=pretrained_weights_wikipedia, trainable=False)(input)
emb2 = Embedding(50000, embedding_size)(input)

bi = Bidirectional(LSTM(embedding_size))

x1 = bi(emb1)
x2 = bi(emb2)

dense = Dense(vocab_size, activation='softmax')

op1 = dense(x1)
op2 = dense(x2)

model = Model(inputs=[input1, input2], outputs=[op1, op2])
model.compile(loss='categorical_crossentropy',
          optimizer = RMSprop(lr=0.0005),
          metrics=['accuracy'])
model.summary()
user_99395420
Related