Validation loss not decreasing using dense layers altough training and validation data have the same distribution

Question

I have a problem that I have great difficulties understanding the concept that leads to these results. I use a keras dense layer to map 13 input features to 3 output labels. During the training, the loss (RMSE) for the training data constantly decreases even after 2000 epochs. However, the loss for the validation data does not decrease any more after some 200 epochs as you can see in the figure.

Now the thing that I don't understand at all is how you can explain this outcome given that the training and validation data have (almost) exactly the same distribution for all of the 13 input features and the 3 output labels. They are drawn from the very same distribution. To illustrate that I have plotted the histograms for some input features and output labels (and the kernel density functions) as you can see here:

If you want to see the histograms for all input features and output labels you can find them here (in all the distribution of the trai, valid and test dataset look quite similar): https://filetransfer.io/data-package/nHxgDfvF#link

Also I calculated the correlation between each of the input features and the output labels for the training, validation and test dataset and the values are almost the same across the datasets for all combinations. If wou want to the see values you can find them here: https://filetransfer.io/data-package/iVruYbLx#link. This just further highlights, that the training and validation data have the same distribution.

Here you can find my code:

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_percentage_error
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import BatchNormalization, Dense, Flatten
from matplotlib import pyplot as plt

#Read data from  csv files
ANN_input_data_features = pd.read_csv("C:/Users/User1/Desktop/TestDataANN_InputFeatures.csv", sep=';')
ANN_input_data_labels = pd.read_csv("C:/Users/User1/Desktop/TestDataANN_OutputLabels.csv", sep=';')
ANN_input_data_features = ANN_input_data_features.values
ANN_input_data_labels = ANN_input_data_labels.values

            
# standardize input features X and output labels Y
scaler_standardized_X = StandardScaler()
ANN_input_data_features = scaler_standardized_X.fit_transform(ANN_input_data_features)

scaler_standardized_Y = StandardScaler()
ANN_input_data_labels = scaler_standardized_Y.fit_transform(ANN_input_data_labels)


#Split dataset into train, validation, an test
index_X_Train_End = int(0.7 * len(ANN_input_data_features))
index_X_Validation_End = int(0.9 * len(ANN_input_data_features))

X_train = ANN_input_data_features [0: index_X_Train_End]
X_valid = ANN_input_data_features [index_X_Train_End: index_X_Validation_End]
X_test = ANN_input_data_features [index_X_Validation_End:]

Y_train = ANN_input_data_labels [0: index_X_Train_End]
Y_valid = ANN_input_data_labels [index_X_Train_End: index_X_Validation_End]
Y_test = ANN_input_data_labels [index_X_Validation_End:]


#Train the model
optimizer_adam = tf.keras.optimizers.Adam(learning_rate= 0.001)

numberOfInputFeatures = len(ANN_input_data_features[0])
numberOfOutputNeurons = len(ANN_input_data_labels[0])

model = keras.Sequential([
    Flatten(input_shape=(numberOfInputFeatures,)),
    Dense(30, activation='relu'),
    #BatchNormalization(axis = 1),
    Dense(50, activation='relu'),
    #BatchNormalization(axis = 1),
    Dense(50, activation='relu'),
    #BatchNormalization(axis = 1),
    Dense(30, activation='relu'),

    keras.layers.Dense(numberOfOutputNeurons)])

entireFolderNameForTheResultsOfTheRun = "C:/Users/User1/Desktop/Training/"
pathOfTheFileForBestModel = entireFolderNameForTheResultsOfTheRun + "bestModelSingleTimeSlotTest.keras"
callbacks = [  keras.callbacks.ModelCheckpoint(pathOfTheFileForBestModel,  save_best_only=True) ]

model.compile(loss="mean_squared_error", optimizer=optimizer_adam, metrics=['mean_absolute_percentage_error'])
history = model.fit(X_train, Y_train, epochs=2000, batch_size=10, validation_data=(X_valid, Y_valid), callbacks=callbacks)

# Predict the values from the test dataset
model = keras.models.load_model(pathOfTheFileForBestModel)
Y_pred = model.predict(X_test)

# Rescale the results of the predictions in the test dataset
Y_test_traInv = scaler_standardized_Y.inverse_transform(Y_test)
Y_pred_traInv = scaler_standardized_Y.inverse_transform(Y_pred)


#  Calculate the error in the test dataset
rms = mean_squared_error(Y_test_traInv, Y_pred_traInv, squared=True)
mape = mean_absolute_percentage_error(Y_test_traInv, Y_pred_traInv)

print("Evaluation with the test data")
print("Root Mean Squarred Error: ", rms)
print("Mean Absolute Percentage Error:", mape)


#Plot training results
plt.plot(history.history['mean_absolute_percentage_error'])
plt.plot(history.history['val_mean_absolute_percentage_error'])
plt.title('Mean absolute percentage errror')
plt.ylabel('Mean absolute percentage errror')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Loss function')
plt.ylabel('Mean squared Error')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()

Here is the data that I use (input features and output labels): https://filetransfer.io/data-package/ldU9KENV#link

I also tried it with batch normalization but this led to overall worse results. And I also played around with the number of layer, neurons, batchsize etc., but the problem remains the same or the results become way worse.

Now my question is how can this be explained. As far as I understand artifical neural networks, the networks adjusts its weight during the training such that it can explain the training dataset as well as possible. But if the training and validation dataset have the same distribution and the same correlation between input and outputs, as in my case, a model capable of explaining the traning dataset should also be able to explain the validation dataset which is not the case here. As the training and validation dataset have the same distribution, overfitting can't be the reason.

user_16424700 · Answer

I went through the training and validation distributions of the 13 features that you have provided. But note that those are marginal distributions of the individual features, not their joint distribution. The term "data distribution" actually means the "joint distribution" of all the features.

First of all, in order to claim that both training data and validation data are drawn from the same distribution, you need to prove that the joint distributions of all features of both training and validation data are same. Because it's quite possible that the marginals of the features may look similar but their joint distributions may be wildly different. You could read more about distributions, joint distributions and marginal distributions. I also suggest reading about Empirical Risk Minimization to understand why (and under what conditions) reducing the loss on training data alone can improve the model's performance on unseen data.

If it so happens that your training and validation data both have the same joint distribution and you are providing sufficient number of training examples to the model, and yet there's a significant gap between validation and training losses, then it might be due to overfitting.

Validation loss not decreasing using dense layers altough training and validation data have the same distribution

1 Answers