Val Loss and manually calculated loss produce different values

Question

I have a CNN classification model that uses loss: binary cross entropy:

optimizer_instance = Adam(learning_rate=learning_rate, decay=learning_rate / 200)
model.compile(optimizer=optimizer_instance, loss='binary_crossentropy')

We are saving the best model so the latest saved model is the one that achieved the best val_loss:

es = EarlyStopping(monitor='val_loss', mode='min', verbose=0, patience=Config.LearningParameters.Patience)
modelPath = modelFileFolder + Config.LearningParameters.ModelFileName
checkpoint = keras.callbacks.ModelCheckpoint(modelPath , monitor='val_loss',
                                                         save_best_only=True,
                                                         save_weights_only=False, verbose=1)
callbacks = [checkpoint,es]
history = model.fit(x=training_generator,
                    batch_size=Config.LearningParameters.Batch_size,
                    epochs=Config.LearningParameters.Epochs,
                    validation_data=validation_generator,                              
                    callbacks=callbacks,
                    verbose=1)

on the course of the training the logs show that the val_loss has decreased to 0.41. At the end of the train we load the best model that was saved during the training process and predicted the validation dataset. Then we calculated the BCE manually and received a totally different value of 2.335.

here is the manual loss calculation:

bce = tf.keras.losses.BinaryCrossentropy()
binaryCSELoss = bce(y_valid, preds)
print("Calculated Val Loss is: " + str(binaryCSELoss ))

here is the end of the training logs:

10/10 [==============================] - ETA: 0s - loss: 0.0778
Epoch 40: val_loss did not improve from 0.41081
10/10 [==============================] - 4s 399ms/step - loss: 0.0778 - val_loss: 0.5413
% of marked 1 in validation: [0.51580906 0.48419094]
% of marked 1 in Test: [0.51991504 0.480085  ]
---------------------------------
Calculated Val Loss is: 2.3350689765791395

We thought that it might have to do something with the face that we are using data generators and the loss is then calculated on the batches individually so we added another test where we do not use data generators:

history = model.fit(x=trainX,y = y_train,
                      epochs=Config.LearningParameters.Epochs,
                      validation_data=(validateion_x,y_valid),
                      callbacks=callbacks,
                      verbose=1)
predictions_cnn = model.predict(validateion_x)
bce = tf.keras.losses.BinaryCrossentropy(from_logits=False)
binaryCSELoss = bce(y_valid, predictions_cnn)
valloss = binaryCSELoss.numpy()
print("binaryCSELoss logits=false on all Val Loss is: " + str(valloss))
bce = tf.keras.losses.BinaryCrossentropy(from_logits=True)
binaryCSELoss = bce(y_valid, predictions_cnn)
valloss = binaryCSELoss.numpy()
print("binaryCSELoss logits=true on all Val Loss is: " + str(valloss))

Here is the end of the training log. Again the loss is no the same:

54/54 [==============================] - ETA: 0s - loss: 0.5015
Epoch 6: val_loss did not improve from 0.66096
54/54 [==============================] - 8s 144ms/step - loss: 0.5015 - val_loss: 1.9742
% of marked 1 in validation: [0.28723404 0.71276593]
% of marked 1 in Test: [0.52077866 0.47922137]
loading Model: E:\CnnModels\2022-06-03_11-53-53\model.h5
Backend TkAgg is interactive backend. Turning interactive mode on.
binaryCSELoss logits=false on all Val Loss is: 0.6353029
binaryCSELoss logits=true on all Val Loss is: 0.7070135

How can this be?

user_7848579 · Answer

BCE is Binary cross entropy when it is responsive to the binary output they mostly understand it as [ 1 -p , p ] that is when we use the output layer with the maximum value for simply representing [ 1 -p , p ]

Sample losses function :

https://towardsdatascience.com/where-did-the-binary-cross-entropy-loss-function-come-from-ac3de349a715

https://www.tensorflow.org/api_docs/python/tf/keras/losses/Loss

The output weights parameters and bias are ( 192, 1 ) representing significance as tree prints and change of the values indicated loss during the time. Loss value is preferred when you read variables value from logs['loss'] for evaluation but for the requirements, mapping result.

[ Sample ]:

import os
from os.path import exists

import tensorflow as tf
import tensorflow_io as tfio

import matplotlib.pyplot as plt

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
None
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
physical_devices = tf.config.experimental.list_physical_devices('GPU')
assert len(physical_devices) > 0, "Not enough GPU hardware devices available"
config = tf.config.experimental.set_memory_growth(physical_devices[0], True)
print(physical_devices)
print(config)

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Variables
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
PATH = os.path.join('F:\datasets\downloads\Actors\train\Pikaploy', '*.tif')
PATH_2 = os.path.join('F:\datasets\downloads\Actors\train\Candidt Kibt', '*.tif')
files = tf.data.Dataset.list_files(PATH)
files_2 = tf.data.Dataset.list_files(PATH_2)

list_file = []
list_file_actual = []
list_label = []
list_label_actual = [ 'Pikaploy', 'Pikaploy', 'Pikaploy', 'Pikaploy', 'Pikaploy', 'Candidt Kibt', 'Candidt Kibt', 'Candidt Kibt', 'Candidt Kibt', 'Candidt Kibt' ]
for file in files.take(5):
    image = tf.io.read_file( file )
    image = tfio.experimental.image.decode_tiff(image, index=0)
    list_file_actual.append(image)
    image = tf.image.resize(image, [32,32], method='nearest')
    list_file.append(image)
    list_label.append(1)
    
for file in files_2.take(5):
    image = tf.io.read_file( file )
    image = tfio.experimental.image.decode_tiff(image, index=0)
    list_file_actual.append(image)
    image = tf.image.resize(image, [32,32], method='nearest')
    list_file.append(image)
    list_label.append(9)

checkpoint_path = "F:\models\checkpoint\" + os.path.basename(__file__).split('.')[0] + "\TF_DataSets_01.h5"
checkpoint_dir = os.path.dirname(checkpoint_path)

if not exists(checkpoint_dir) : 
    os.mkdir(checkpoint_dir)
    print("Create directory: " + checkpoint_dir)

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
DataSet
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
dataset = tf.data.Dataset.from_tensor_slices((tf.constant(tf.cast(list_file, dtype=tf.int64), shape=(10, 1, 32, 32, 4), dtype=tf.int64),tf.constant(list_label, shape=(10, 1, 1), dtype=tf.int64)))

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: Model Initialize
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
model = tf.keras.models.Sequential([
    tf.keras.layers.InputLayer(input_shape=( 32, 32, 4 )),
    tf.keras.layers.Normalization(mean=3., variance=2.),
    tf.keras.layers.Normalization(mean=4., variance=6.),
    tf.keras.layers.Conv2D(32, (3, 3), activation='relu'),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Reshape((128, 225)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(96, return_sequences=True, return_state=False)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(96)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(192, activation='relu'),
    tf.keras.layers.Dense(1, name='output'),
])

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: Callback
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
class custom_callback(tf.keras.callbacks.Callback):
        
    def on_epoch_end(self, epoch, logs=None):
        if( logs['accuracy'] >= 0.97 ):
            self.model.stop_training = True
            
        print( "% of marked 2 in Train:  " + str( self.model.get_layer( name='output' ).get_weights()[0][ tf.math.argmax( self.model.get_layer( name='output' ).get_weights()[0] ).numpy() ][0][0] ) + " " + str( 1 - self.model.get_layer( name='output' ).get_weights()[0][ tf.math.argmax( self.model.get_layer( name='output' ).get_weights()[0] ).numpy() ][0][0] )
        )

    def on_test_end(self, logs=None):
        print( "
" )   
        print( "% of marked 1 in Train:  " + str( self.model.get_layer( name='output' ).get_weights()[0][ tf.math.argmax( self.model.get_layer( name='output' ).get_weights()[0] ).numpy() ][0][0] ) + " " + str( 1 - self.model.get_layer( name='output' ).get_weights()[0][ tf.math.argmax( self.model.get_layer( name='output' ).get_weights()[0] ).numpy() ][0][0] )
        )
        # print( "
" )     
    
custom_callback = custom_callback()

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: Optimizer
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
optimizer = tf.keras.optimizers.Nadam(
    learning_rate=0.00001, beta_1=0.9, beta_2=0.999, epsilon=1e-07,
    name='Nadam'
)

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: Loss Fn
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""                               
lossfn = tf.keras.losses.BinaryCrossentropy( 
    from_logits=False,
    reduction=tf.keras.losses.Reduction.AUTO,
    name='BinaryCrossentropy' )

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: Model Summary
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
model.compile(optimizer=optimizer, loss=lossfn, metrics=['accuracy'])

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: FileWriter
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
if exists(checkpoint_path) :
    model.load_weights(checkpoint_path)
    print("model load: " + checkpoint_path)
    input("Press Any Key!")

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: Training
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
history = model.fit( dataset, validation_data=(dataset), batch_size=1, epochs=50, callbacks=[custom_callback] )
model.save_weights(checkpoint_path)

plt.figure(figsize=(5,2))
plt.title("Actors recognitions")
for i in range(len(list_file)):
    img = tf.keras.preprocessing.image.array_to_img(
        list_file[i],
        data_format=None,
        scale=True
    )
    img_array = tf.keras.preprocessing.image.img_to_array(img)
    img_array = tf.expand_dims(img_array, 0)
    predictions = model.predict(img_array)
    score = tf.nn.softmax(predictions[0])
    plt.subplot(5, 2, i + 1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(list_file_actual[i])
    plt.xlabel(str(round(score[tf.math.argmax(score).numpy()].numpy(), 2)) + ":" +  str(list_label_actual[tf.math.argmax(score)]))
    
plt.show()

input('...')

[ Output ]:

10/10 [==============================] - 1s 56ms/step - loss: -60.9311 - accuracy: 0.5000 - val_loss: -60.9329 - val_accuracy: 0.5000
Epoch 6/50
 9/10 [==========================>...] - ETA: 0s - loss: -54.1486 - accuracy: 0.5556

% of marked 1 in Train:  0.17788188 0.8221181184053421
% of marked 2 in Train:  0.17788188 0.8221181184053421
10/10 [==============================] - 1s 54ms/step - loss: -60.9331 - accuracy: 0.5000 - val_loss: -60.9341 - val_accuracy: 0.5000
Epoch 7/50
 9/10 [==========================>...] - ETA: 0s - loss: -54.1499 - accuracy: 0.5556

% of marked 1 in Train:  0.17788248 0.8221175223588943
% of marked 2 in Train:  0.17788248 0.8221175223588943
10/10 [==============================] - 1s 57ms/step - loss: -60.9343 - accuracy: 0.5000 - val_loss: -60.9351 - val_accuracy: 0.5000
Epoch 8/50
 9/10 [==========================>...] - ETA: 0s - loss: -54.1509 - accuracy: 0.5556

% of marked 1 in Train:  0.1778828 0.8221171945333481
% of marked 2 in Train:  0.1778828 0.8221171945333481

[ Output ]:

Val Loss and manually calculated loss produce different values

1 Answers