How do you make predictions with a stateful LSTM?

user_1172541 0
Viewed 499

Okay, so I trained a stateful LSTM characterwise on https://cs.stanford.edu/people/karpathy/char-rnn/shakespear.txt. It didn't seem to do too bad in terms of accuracy, but know I want to generate my own shakespeare works.

The question is, how do I go about actually generating predictions from it?

In particular, the models batch input shape is (128, 128, 63) and the output shape is (128, 128, 63). (The first number is the batch size, the second number is the length of the prediction input and output, and the third number is the number of distinct characters in the text.)

For example, I would like to:

  • Generate various predictions starting from empty text
  • Generate predictions starting from a small starting text (such as "PYRULEZ:")

This should be possible given how LSTMs work.

Here's a snippet of the code used to generate and fit the model:

model = Sequential()
model.add(LSTM(dataY.shape[2], batch_input_shape=(128, dataX.shape[1], dataX.shape[2]), return_sequences = True, stateful=True, activation = "softmax"))
model.summary()
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics = ['acc'])
model.fit(dataX, dataY, epochs = 1, batch_size = 128, verbose=1, shuffle = False)

Looking at other code samples, it appears I'll need to modify this somehow, but I'm not sure in how specifically.

I can include the whole code sample if that would be helpful. It is self contained.

lstm keras tensorflow prediction python
1 Answers

Simple. Put your input into model.predict() with appropriate parameters (see documentation), concatenate input and output (the model predicts on progressively longer chains). Depending on how you organised training, output will add one character at a time. To be more precise, if you train sequence to sequence shifted by one, your output sequence will ideally be your input sequence shifted by one element; PYRULEZ -> YRULEZ* Hence you need to take the last character of the output and add it to your prior (input) sequence.

If you want long lines of text, you might want to limit the length of your sequence to some number of characters in the loop. Much of the long term dependencies in the text is carried through the stateful vector of the LSTM cell anyway (Not something you interact with).

Pseudocode-ish:

for counter in range(output_length):
    output = model.predict(input_)
    input_ = np.concatenate((input_, output[:,-1,:]), axis=1)
user_93900790
Related