Higher validation accuracy, than training accurracy using Tensorflow and Keras

user_2200569 0
Viewed 70026

I'm trying to use deep learning to predict income from 15 self reported attributes from a dating site.

We're getting rather odd results, where our validation data is getting better accuracy and lower loss, than our training data. And this is consistent across different sizes of hidden layers. This is our model:

for hl1 in [250, 200, 150, 100, 75, 50, 25, 15, 10, 7]:
    def baseline_model():
        model = Sequential()
        model.add(Dense(hl1, input_dim=299, kernel_initializer='normal', activation='relu', kernel_regularizer=regularizers.l1_l2(0.001)))
        model.add(Dropout(0.5, seed=seed))
        model.add(Dense(3, kernel_initializer='normal', activation='sigmoid'))

        model.compile(loss='categorical_crossentropy', optimizer='adamax', metrics=['accuracy'])
        return model

    history_logs = LossHistory()
    model = baseline_model()
    history = model.fit(X, Y, validation_split=0.3, shuffle=False, epochs=50, batch_size=10, verbose=2, callbacks=[history_logs])

And this is an example of the accuracy and losses: Accuracy with hidden layer of 250 neurons and the loss.

We've tried to remove regularization and dropout, which, as expected, ended in overfitting (training acc: ~85%). We've even tried to decrease the learning rate drastically, with similiar results.

Has anyone seen similar results?

keras tensorflow classification machine-learning neural-network
8 Answers

There are a number of reasons this can happen.You do not shown any information on the size of the data for training, validation and test. If the validation set is to small it does not adequately represent the probability distribution of the data. If your training set is small there is not enough data to adequately train the model. Also your model is very basic and may not be adequate to cover the complexity of the data. A drop out of 50% is high for such a limited model. Try using an established model like MobileNet version 1. It will be more than adequate for even very complex data relationships. Once that works then you can be confident in the data and build your own model if you wish. Fact is validation loss and accuracy do not have real meaning until your training accuracy gets reasonably high say 85%.

user_107989170

I solved this by simply increasing the number of epochs

user_127985880

Adding dropout to your model gives it more generalization, but it doesn't have to be the cause. It could be because your data is unbalanced (has bias) and that's what I think..

user_160102860

I don't think that it is a drop out layer problem.

I think that it is more related to the number of images in your dataset.

The point here is that you are working on a large training Set and a too small validation/test set so that this latter is way too easy to computed.

Try data augmentation and other technique to get your dataset bigger!

user_174208920
Related