Using classification_report to evaluate a Keras model

Question

The Problem:

During training the performance of my model looks quite allright. However, the results of the classification_report from sklearn yields a precision, recall and f1 of zero almost everywhere. What am I doing wrong to get such a missmatch between training performance and inference? (I am using Keras with a TensorFlow backend.)

My code:

I use the valiation_split argument to generate two generators (train, validation) like so:

train_datagen = ImageDataGenerator(
rescale=1. / 255, validation_split=0.15)

train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='categorical', subset="training")

validation_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='categorical', subset="validation", shuffle=False)

I set shuffle=False in my validation_generator to make sure it does not mix the relationship of images and labels for my evaluation later on.

Next, I train my model like so:

history = model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size,
    verbose=1)

Performance is allright:

Epoch 1/5
187/187 [==============================] - 44s 233ms/step - loss: 0.7835 - acc: 0.6744 - val_loss: 1.2918 - val_acc: 0.6079
Epoch 2/5
187/187 [==============================] - 42s 225ms/step - loss: 0.7578 - acc: 0.6901 - val_loss: 1.2962 - val_acc: 0.6149
Epoch 3/5
187/187 [==============================] - 40s 216ms/step - loss: 0.7535 - acc: 0.6907 - val_loss: 1.3426 - val_acc: 0.6061
Epoch 4/5
187/187 [==============================] - 41s 217ms/step - loss: 0.7388 - acc: 0.6977 - val_loss: 1.2866 - val_acc: 0.6149
Epoch 5/5
187/187 [==============================] - 41s 217ms/step - loss: 0.7282 - acc: 0.6960 - val_loss: 1.2988 - val_acc: 0.6297

Now, I extract the necessary info for the classification_report following the method suggested here https://github.com/keras-team/keras/issues/2607#issuecomment-302365916 . This gives me the following:

validation_steps_per_epoch = np.math.ceil(validation_generator.samples / validation_generator.batch_size)

predictions = model.predict_generator(validation_generator, steps=validation_steps_per_epoch)
# Get most likely class
predicted_classes = np.argmax(predictions, axis=1) 

true_classes = validation_generator.classes
class_labels = list(validation_generator.class_indices.keys())

Finally, I output the classification report using:

from sklearn.metrics import classification_report
report = classification_report(true_classes, predicted_classes, target_names=class_labels)
print(report)

Which results in zeros all over the place (see avgs. below):

                   precision    recall  f1-score   support
      micro avg       0.01      0.01      0.01      2100
      macro avg       0.01      0.01      0.01      2100
   weighted avg       0.01      0.01      0.01      2100

Using classification_report to evaluate a Keras model

0 Answers