Keras image classification: High accuracy shown but low on test images

user_4146344 0
Viewed 1207

I'm trying to do some image classification on the Caltech101 dataset. I used several pretrained models in Keras. I used some augmentation on the training set:

train_datagen = keras.preprocessing.image.ImageDataGenerator(
                    rescale=1./255, rotation_range=15,
                                   width_shift_range=0.1,
                                   height_shift_range=0.1,
                                   shear_range=0.01,
                                   zoom_range=[0.9, 1.25],
                                   horizontal_flip=False,
                                   vertical_flip=False,
                                   fill_mode='reflect',
                                   data_format='channels_last',
                                   brightness_range=[0.5, 1.5])
    validation_datagen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255)

    train_generator = train_datagen.flow_from_directory(
                    train1_dir,  # Source directory for the training images
                    target_size=(image_size, image_size),
                    batch_size=batch_size)

    validation_generator = validation_datagen.flow_from_directory(
                    validation_dir, # Source directory for the validation images
                    target_size=(image_size, image_size),
                    batch_size=batch_size)

I also used some early stopping (stop after 100 epochs):

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=100)
    mc = ModelCheckpoint('best_model_%s_%s.h5' % (dataset_name, model_name), monitor='val_acc', mode='max', verbose=1, save_best_only=True)
    callbacks = [es, mc]

At first I train the last layer:

base_model.trainable = False
    model = tf.keras.Sequential([
      base_model,
      keras.layers.GlobalAveragePooling2D(),
      keras.layers.Dense(num_classes, activation='softmax')
    ])
    model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.0001),
                  loss='categorical_crossentropy',
                  metrics=['accuracy'])
    epochs = 10000
    steps_per_epoch = train_generator.n // batch_size
    validation_steps = validation_generator.n // batch_size
    history = model.fit_generator(train_generator,
                                  steps_per_epoch = steps_per_epoch,
                                  epochs=epochs,
                                  workers=4,
                                  validation_data=validation_generator,
                                  validation_steps=validation_steps, 
                                  callbacks=callbacks)

Then I train the previous layers, following the Keras tutorial:

# After top classifier is trained, we finetune the layers of the network
base_model.trainable = True
# Let's take a look to see how many layers are in the base model
print("Number of layers in the base model: ", len(base_model.layers))
# Fine tune from this layer onwards
fine_tune_at = 1
# Freeze all the layers before the `fine_tune_at` layer
for layer in base_model.layers[:fine_tune_at]:
    layer.trainable =  False

model.compile(optimizer = tf.keras.optimizers.RMSprop(lr=2e-5),
              loss='categorical_crossentropy',
              metrics=['accuracy'])
epochs = 10000
history_fine = model.fit_generator(train_generator,
                                   steps_per_epoch = steps_per_epoch,
                                   epochs=epochs,
                                   workers=4,
                                   validation_data=validation_generator,
                                   validation_steps=validation_steps, 
                                   callbacks=callbacks
                                   )

Finally, after the model has finished training, I test it manually on a separate test set

label_list = train_generator.class_indices
    numeric_to_class = {}
    for key, val in label_list.items():
        numeric_to_class[val] = key
    total_num_images = 0
    acc_num_images = 0
    with open("%s_prediction_%s.txt" % (dataset_name, model_name), "wt") as fid:
        fid.write("Label list:\n")
        for label in label_list:
            fid.write("%s," % label)
        fid.write("\n")
        fid.write("true_class,predicted_class\n")
        fid.write("--------------------------\n")
        for label in label_list:
            testing_dir = os.path.join(test_dir, label)
            for img_file in os.listdir(testing_dir):
                img = cv2.imread(os.path.join(testing_dir, img_file))
                img_resized = cv2.resize(img, (image_size, image_size), interpolation = cv2.INTER_AREA)                
                img1 = np.reshape(img_resized, (1, img_resized.shape[0], img_resized.shape[1], img_resized.shape[2]))
                pred_class_num = model.predict_classes(img1)
                pred_class_num = pred_class_num[0]
                true_class_num = label_list[label]                
                predicted_label = numeric_to_class[pred_class_num]               
                fid.write("%s,%s\n" % (label, predicted_label))
                if predicted_label == label:
                    acc_num_images += 1
                total_num_images += 1

    acc = acc_num_images / (total_num_images * 1.0)

I had to do this because the library does not output F1 score. However what I found out is that the val_acc goes up very high (about 0.8) but during the test phase after training the accuracy is very low (about 0.1 I think). I do not understand why this is so. Please help me, thank you very much.

UPDATE 15/10/2019: I tried to just train a linear svm on top of the network without finetuning anything and I got 70% accuracy on Caltech101 using VGG16 (with RMSProp optimizer). However I'm not sure if this is the best choice.

UPDATE 2: I used the preprocessing part suggested by Daniel Moller on my custom dataset (about 450 images, 283 class "open", 203 class "closed", and got this accuracy and loss when using early stopping with patience=100, just training the last layer with:

model = tf.keras.Sequential([
  base_model,
  keras.layers.GlobalAveragePooling2D(),
  keras.layers.Dense(num_classes, activation='softmax')
])

Accuracy

Loss

UPDATE 3: I tried to use the last fully connected layers in VGG16 also, and added dropout layer after each of them, with the dropout rate (the rate that is set to 0) of 60%, and patience=10 (for early stopping):

base_model = tf.keras.applications.VGG16(input_shape=IMG_SHAPE, \
                                               include_top=True, \
                                               weights='imagenet')
base_model.layers[-3].Trainable = True
base_model.layers[-2].Trainable = True
fc1 = base_model.layers[-3]
fc2 = base_model.layers[-2]
predictions = keras.layers.Dense(num_classes, activation='softmax')
dropout1 = Dropout(0.6)
dropout2 = Dropout(0.6)
x = dropout1(fc1.output)
x = fc2(x)
x = dropout2(x)
predictors = predictions(x)
model = Model(inputs=base_model.input, outputs=predictors)

And I got the highest validation accuracy of 0.93750, test accuracy: 0.966216. Graphs: enter image description here Accuracy

keras tensorflow machine-learning python
1 Answers

Main issue seems to be here:

You forgot to rescale 1/255. when you open the images for prediction in:

.....
img = cv2.imread(os.path.join(testing_dir, img_file))
img_resized = cv2.resize(img, (image_size, image_size), interpolation = cv2.INTER_AREA)
img1 = np.reshape(img_resized, 
                  (1, img_resized.shape[0], img_resized.shape[1], img_resized.shape[2]))

#you will probably need:
img1 = img1/255.

pred_class_num = model.predict_classes(img1)
...........

Also, notice that cv2 will open images in BGR format, while Keras may be opening them in RGB.

  • Get images from the Keras generator
  • Get images opened by you
  • Plot these images to check if they look ok (or at least the same, if everything is BGR, it's not a problem although all plots will look funny)

Example:

keras_train = train_generator[0][0] #first image from first batch
keras_val = validation_generator[0][0]

img = cv2.imread(os.path.join(testing_dir, img_file))
img_resized = cv2.resize(img, (image_size, image_size), interpolation = cv2.INTER_AREA) 
img1 = np.reshape(img_resized, 
                  (1, img_resized.shape[0], img_resized.shape[1], img_resized.shape[2]))    
your_image = img1[0]/255. #first image from your batch rescaled

Plot these images with matplotlib.
Also make sure they have the same range:

plt.imshow(keras_train)
plt.plot()
plt.imshow(keras_val)
plt.plot()
plt.imshow(your_image)
plt.plot()
print(keras_train.max(), keras_val.max(), img1.max())

You may need to use np.flip(images, axis=-1) to transform BGR into RGB.

Hints for importing keras models

If you imported base model from keras, you should import the preprocessing from the same module, as well as use Keras image opener. This would eliminate possible mismatches:

from keras.applications.resnet50 import ResNet50
from keras.preprocessing import image #use this instead of cv2   
from keras.applications.resnet50 import preprocess_input #use this in the generators

In both generators, use the preprocessing function:

#no rescale, only preprocessing function
train_datagen = keras.preprocessing.image.ImageDataGenerator(
                               rotation_range=15,
                               width_shift_range=0.1,
                               height_shift_range=0.1,
                               shear_range=0.01,
                               zoom_range=[0.9, 1.25],
                               horizontal_flip=False,
                               vertical_flip=False,
                               fill_mode='reflect',
                               data_format='channels_last',
                               brightness_range=[0.5, 1.5],
                               preprocessing_function=preprocess_input)
validation_datagen = keras.preprocessing.image.ImageDataGenerator(
                               preprocessing_function=preprocess_input)

Load and preprocess your images for prediction:

img = image.load_img(img_path, target_size=(image_size,image_size))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

More in: https://keras.io/applications/

Other possibilities

But there could be several other things like:

  • Very strong overfitting, your patience for the early stopper is 100! Usually this is a very high number. But the only way to confirm this is to check whether the val_acc is rising too much compared with the best one.
    • Suggestion: reload the best model saved by ModelCheckpoint before you start predicting
    • Suggestion: post the acc history so we can see whether it's bad
  • Does the base model have BatchNormalization layers? - When you freeze a batch normalization layer, it will keep the moving_mean and moving_variance unchanged, but these values were trained with a different data. If your data doesn't have the same mean and variance, your model will train well, but validation will be a disaster
    • Suggestion: see whether the validation loss/acc is terribly wrong since the beginning
    • Suggestion: don't freeze the BatchNormalization layers in the base model (iterate over the layers and freeze only the other types)
    • Suggestion: if you happen to know the database with which the base model was trained, compare your data mean and variation with that database's mean and variation (treat each channel individually)
user_20972400
Related