Does Keras's LSTM really take into account the cell state and previous output?

user_8585864 0
Viewed 502

I learned about LSTM's over the past day, and then i decided to look at a tutorial which uses Keras to create it. I looked at several tutorials and they all had a derivative of 

model = Sequential()
model.add(LSTM(10, input_shape=(1,1)))
model.add(Dense(1, activation='linear'))
model.compile(loss='mse', optimizer='adam')
X,y = get_train()
model.fit(X, y, epochs=300, shuffle=False, verbose=0)

then they predicted using

 model.predict(X, verbose=0)

my question is: don't you have to give the previous prediction along with input and cell state in order to predict the next outcome using an LSTM? Also, what does the 10 represent in model.add(LSTM(10, input_shape(1,1))?

lstm keras tensorflow python
3 Answers

You have to give the previous prediction to the LSTM state. If you call predict the LSTM will be initialized every time, it will not remember the state from previous predictions.

Typically (e.g if you generate text with an lstm) you have a loop where you do something like this:

# pick a random seed
start = numpy.random.randint(0, len(dataX)-1)
pattern = dataX[start]
print "Seed:"
print "\"", ''.join([int_to_char[value] for value in pattern]), "\""
# generate characters
for i in range(1000):
    x = numpy.reshape(pattern, (1, len(pattern), 1))
    x = x / float(n_vocab)
    prediction = model.predict(x, verbose=0)
    index = numpy.argmax(prediction)
    result = int_to_char[index]
    seq_in = [int_to_char[value] for value in pattern]
    sys.stdout.write(result)
    pattern.append(index)
    pattern = pattern[1:len(pattern)]
print "\nDone."

(example copied from machinelearningmastery.com)

The important thing are this lines:

pattern.append(index)
pattern = pattern[1:len(pattern)]

Here they append the next character to the pattern and then drop the first character to have an input length that matches the expectation from the lstm. Then the bring it to a numpy array (x = np.reshape(...)) and predict from the model with the generated output. So to answer your first question you need to feed in the output again.

For the second question the 10 corresponds to the number of lstm cells that you have in a layer. If you don't use "return_sequences=True" it corresponds to the output size of that layer.

user_56320580

Let's break it down into pieces and look pictorially

LSTM(10, input_shape=(3,1))): Defines an LSTM whose sequence length is 3 i.e the LSTM will unroll for 3 timesteps. At each timestep the LSTM will take an input of size 1. The output (and also the size of the hidden state and all other LSTM gates) is 10 (vector or size 10)

![enter image description here

You dont have to do unrolling manually (passing in the current hidden state to the next state) it is taken care by the keras/tensorflow LSTM layer. All you have to do is to pass in data in the (batch_size X time_steps X input_size) format.

Dense(1, activation='linear'): This is a dense layer with linear activation with takes in as input the output of the previous layer (i.e the output of the LSTM which will be a vector of size 10 of the last unrolling). It will return a vector of size 1.

enter image description here

The same can be checked using model.summary()

user_4239260

Your 1st question:

don't you have to give the previous prediction along with input and cell state in order to predict the next outcome using an LSTM?

no, you don't have to do that. As far as I understand, it is stored in the LSTM cell which is why LSTM uses so much RAM

if you have data with shape looking like this:

(100,1000)

if you plug that into the fit function, each epoch will run on 100 lists. The LSTM will remember 1000 data plots before refreshing when it moves onto the next list.

2nd:

Also, what does the 10 represent in model.add(LSTM(10, input_shape(1,1))?

it is the shape of the 1st layer after the input, so your model currently has the shape of:

1,1 10 1

hope it helps :)

user_112190430
Related