I was working on a network that counts the number of sea lions in drone images for a Kaggle competition, which just ended. The sea lions are also differentiated by class (adult male, adult female, etc). Please help me understand why adding dropout to this network did not work.
The network architecture in Keras is copied below. Initially, I took the first fifteen layers of VGG16 pretrained on Imagenet. On top of those I put seven layers: Convolutional, RELU, Pooling, Convolutional, RELU, Pooling, and global average pooling to output the counts. This model overfit the data during training, ending with a loss of 400 or 500 (excellent scores) but a holdout validation loss of ~850 (decent). Then, I added 50% dropout after the VGG layers and again after each of the layers I added. This was per the Stanford cs231 online course notes - "It is also common to combine this with dropout applied after all layers. The value of p=0.5 is a reasonable default." After training this for a while, training loss had come down to ~800 but holdout validation loss was still stuck at 1200+ (not good at all), which it had reached much earlier in the training.
Why didn't adding dropout work? Is something else such as the architecture, the task, or the optimizer somehow the issue?
Edit: Also, there are only ~800 training images and another ~150 in the holdout validation set, but I was also using data augmentation with horizontal flip, vertical flip, and up to 180 degrees of rotation.
base_model = VGG16(weights='imagenet', include_top=False)
x = base_model.layers[15].output
x = Dropout(0.5)(x)
x = Conv2D(128, kernel_size=(3, 3), padding='same')(x)
x = Dropout(0.5)(x)
x = Activation('relu')(x)
x = Dropout(0.5)(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Dropout(0.5)(x)
x = Conv2D(n_classes, kernel_size=(3, 3), padding='same')(x)
x = Dropout(0.5)(x)
x = Activation('relu')(x)
x = Dropout(0.5)(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Dropout(0.5)(x)
x = GlobalAveragePooling2D()(x)
model = Model(inputs=base_model.input, outputs=x)
for layer in base_model.layers:
layer.trainable = False
model.compile(loss=keras.losses.mean_squared_error,
optimizer= keras.optimizers.Adadelta())