Huge decrease in val_acc in my model training, what is the reason?

user_13099218 0
Viewed 517

I'm training CAT/DOG classifier.

My model is:

model.add(layers.Conv2D(32, (3, 3), activation='relu',
                        input_shape=(150, 150, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
            optimizer=optimizers.RMSprop(lr=1e-4),
            metrics=['acc'])
history = model.fit_generator(
                    train_generator,
                    steps_per_epoch = 100,
                    epochs=200,
                    validation_data=validation_generator,
                    validation_steps=50)

My val_acc ~83%, my val_loss ~0.36 between 130th-140th epoch - excluding 136th epoch -.

Epoch 130/200
100/100 [==============================] - 69s - loss: 0.3297 - acc: 0.8574 - val_loss: 0.3595 - val_acc: 0.8331
Epoch 131/200
100/100 [==============================] - 68s - loss: 0.3243 - acc: 0.8548 - val_loss: 0.3561 - val_acc: 0.8242
Epoch 132/200
100/100 [==============================] - 71s - loss: 0.3200 - acc: 0.8557 - val_loss: 0.2725 - val_acc: 0.8157
Epoch 133/200
100/100 [==============================] - 71s - loss: 0.3236 - acc: 0.8615 - val_loss: 0.3411 - val_acc: 0.8388
Epoch 134/200
100/100 [==============================] - 70s - loss: 0.3115 - acc: 0.8681 - val_loss: 0.3800 - val_acc: 0.8073
Epoch 135/200
100/100 [==============================] - 70s - loss: 0.3210 - acc: 0.8536 - val_loss: 0.3247 - val_acc: 0.8357

Epoch 137/200
100/100 [==============================] - 66s - loss: 0.3117 - acc: 0.8602 - val_loss: 0.3396 - val_acc: 0.8351
Epoch 138/200
100/100 [==============================] - 70s - loss: 0.3211 - acc: 0.8624 - val_loss: 0.3284 - val_acc: 0.8185

I wonder why this happened in 136th epoch, val_loss raised to 0.84:

Epoch 136/200
100/100 [==============================] - 67s - loss: 0.3061 - acc: 0.8712 - val_loss: 0.8448 - val_acc: 0.6881

It was an extremely unlucky dropout at that dropped all important values from activation matrix or what?

Here is my final result:

i.stack.imgur.com/BScxe.png

How model is able to solve this?

Thank you :)

keras tensorflow deep-learning neural-network artificial-intelligence
2 Answers

The architecture that you are using is somehow similar to a VGG.

This sudden drop that you are experimenting is due to the fact that your model simply starts to strongly overfit after that epoch.

An additional observation , out of personal experience, is that sudden/such a huge discrepancy between training and validation , at an advanced step during the training, takes place on networks which do not have skip-connections. Note that this phenomenon that I am referring to is different than the 'mere' overfitting.

Networks which do have skip-connections do not exhibit this sudden huge drop phenomenon(particularly at an advanced step during the training phase). The main intuition is that by means of those skip connection information flow of the gradient is not lost. However, on a very deep convolutional neural network which does not have such connections, you can arrive at a point where you have a sudden drop (even both on training accuracy, because of the vanishing gradient).

For more about skip/residual connections, read more about here : https://www.quora.com/How-do-skip-connections-work-in-a-fully-convolutional-neural-network.

UPDATE (according to the photos uploaded):

The sudden drop is only caused by the batch training(hopefully you are not in the case I described above). When we use batch training (since we do not have enough memory to fit the entire dataset at once). Fluctuations are normal, it just happened that at that specific epoch the weights had such values in that the accuracy decreased a lot. Indeed, decreasing the learning rate would help you obtain better accuracy and validation accuracy, since it would help the neural network 'exit' a possible state of plateau.

user_61170170

It is normal for the values to fluctuate. In your case, it can be explained by the value of your learning rate and a large number of epochs.

You are training for too long, you have reached a plateau (accuracy isn't improving). Using big learning rates at the end of training can cause plateauing or convergence issues.

learning rate vs validation accuracy

In the image, you can see that for learning rate=0.1 it reaches high accuracy very fast but then plateaus and drops in accuracy. For a learning rate=0.001, it reaches high accuracy slower but is continuously increasing.

So in your case, I think the problem is the large learning rate towards the endo of the training. You can use a variable learning rate to get the best of both worlds, big at first but lower towards the end. For example, after the accuracy isn't increasing more than 0.1% drop learning rate to 0.0000001.

You can do this using LearningRateScheduler or ReduceLROnPlateau from keras callbacks

reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
                          patience=5, min_lr=1e-10)
model.fit_generator(
    train_generator,
    steps_per_epoch = 100,
    epochs=200,
    validation_data=validation_generator,
    validation_steps=50,
    callbacks=[reduce_lr])
user_63706550
Related