Predicting in Keras with LSTM layer

user_9756652 0
Viewed 835

I'm trying to build a text labeling (multilabel) neural network using Keras.

I have built a dictionary of about 2000 words, and encoded training samples as sequeces of word indices of length 140 (with padding).

As the result data looks like a 2D array of size (num_samples, 140). Where number of samples is around 30k.

Here is the definition on my neural network

mdl = Sequential()
mdl.add(Embedding((vocab_len + 1), 300, input_length=140))
mdl.add(LSTM(100))

mdl.add(Dense(train_y.shape[1], activation="sigmoid"))

mdl.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=["accuracy"])

history = mdl.fit(train_x, train_y, epochs=4, verbose=1, validation_data=(valid_x, valid_y), batch_size=100)

During training the Keras shows accuracy around 0.93 on both trainig and validation data. Which looks promising.

But when I try to invoke predict on test data

pred_y = mdl.predict(test_x, batch_size=100)

I get an array, where all rows look identical, and all less than 0.5. Hence no labels is set on any of test samples.

Sample output from mdl.predict()

The same behaviour is observed if I run predict() on the very same training data I just used to train the model.

But if I run mdl.evaluate() I get the same accuracy of 0.93 as shown during model fitting.

What am I doing wrong?

rnn lstm keras python
2 Answers

Accuracy is not good metrics if your classes are unbalanced. Imaging you have a dataset with 80% of 0's and 20% of 1's. You can create a model which will return 0 in all cases and it's accuracy will be equal to 80%.

user_14124030

model.predict(test_x, batch_size=100) will output the probability of each of the classes

model.predict_classes(test_x, batch_size=100) will output the most probable class/actual prediction

So from your question it seems you want model.predict_classes. Run dir(model) to see all available functions.

If you want to generate model.predict_classes output from model.predict output, do 

pred = model.predict()
pred_classes_output = pred.argmax(axis=1)

what this does is, it goes over every row for example below is output of model.predict which contains probability for class1, class2, class3

[[0.15, 0.73, 0.02], # note the sum of probabilities is = 1

[0.23, 0.33, 0.44]]

it finds the index of the maximum probability i.e 0.73 has index 1, and it creates an array of that so the output will be [1, 2].

model.evaluate runs the check on model.predict_classes.

Also I hope you understand that if your sample was inherenly biased, say you had 90 of class 1, 10 of class 2, then by just predicting 1 it will get 90% accuracy known as baseline accuracy.

user_37374680
Related