The Model
My model is constucted as such:
- Embedding(input_dim=VOCABULARY_SIZE, output_dim=50, input_length=49))
- LSTM(256)
- Dropout(0.2)
- LSTM(256)
- Dropout(0.2)
- Dense(VOCABULARY_SIZE, activation='softmax')
The Dataset
I've created a dataset that contains many articles from Wikipedia which are separated into sections of 49 words and a target word which is the 50th word.
For example:
| Sentence | Target |
|---|---|
| the south michigan avenue historic district is a primarily residential historic district located along south michigan avenue between lyon and lee streets in saginaw michigan it was listed on the national register of historic places in 1982 history development of this neighborhood began in the 1870s and progressed slowly | and |
| south michigan avenue historic district is a primarily residential historic district located along south michigan avenue between lyon and lee streets in saginaw michigan it was listed on the national register of historic places in 1982 history development of this neighborhood began in the 1870s and progressed slowly and | evenly |
Pre Processing
I passed each line in TensorFlow's tokenizer (where the line is separated by spaces so each word is a token, and each word is in lower-caps), then created a generator (to lower RAM usage) that generates one line at a time, where each line (X) passes the tokenizer's texts_to_sequences method, and each target (Y) passes TensorFlow's to_categorical method where the num_class equals the tokenizer's word index length + 1.
The Training
I trained the model over 160000 sequential lines as mentioned above, divided into sets of 160 batches over 100 epochs.
During training my model's accuracy and loss change frequently and not improving, presented here:
How can I improve my model's accuracy?