How can I call a test set at the end of each epoch during the training? I am using tensorflow

user_11849691 0
Viewed 321

I am using Tensorflow-Keras to develop a CNN model in which I have split the data set into train, validation, and test sets. I need to call the test set at the end of each epoch as well as train and validation sets to evaluate the model performance. Below is my code to track train and validation sets.

result_dic = {"epochs": []}
json_logging_callback = LambdaCallback(
                on_epoch_begin=lambda epoch, logs: [learning_rate],
                on_epoch_end=lambda epoch, logs:
                result_dic["epochs"].append({
                    'epoch': epoch + 1, 
                    'acc': str(logs['acc']), 
                    'val_acc': str(logs['val_acc'])
                }))
model.fit(x_train, y_train,
                      validation_data=(x_val, y_val),
                      batch_size=batch_size,
                      epochs=epochs,
                      callbacks=[json_logging_callback])

output:

Epoch 1/5
1/1 [==============================] - 4s 4s/step - acc: 0.8611 - val_acc: 0.8333

However, I'm not sure how to add the test set to my callback to produce the following output.

Expected output:

Epoch 1/5
1/1 [==============================] - 4s 4s/step - acc: 0.8611 - val_acc: 0.8333  - test_acc: xxx
conv-neural-network keras tensorflow python
1 Answers

To display your test accuracy after each epoch, you could customize your fit function to display this metric. Check out this documentation or you could, as shown here, define a simple callback for your test dataset and pass it into your fit function:


model.fit(x_train, y_train,
                      validation_data=(x_val, y_val),
                      batch_size=batch_size,
                      epochs=epochs,
                      callbacks=[json_logging_callback, 
                                 your_test_callback((X_test, Y_test))])

If you want complete flexibility, you could try using a training loop.

Update: Since you want to have a single JSON for all metrics, you should do the following:

Define your TestCallBack and add your test accuracy (and loss if you want) to your logs dictionary:

import tensorflow as tf

class TestCallback(tf.keras.callbacks.Callback):
    def __init__(self, test_data):
        self.test_data = test_data

    def on_epoch_end(self, epoch, logs):
        x, y = self.test_data
        loss, acc = self.model.evaluate(x, y, verbose=0)
        logs['test_accuracy'] = acc

Then add the test accuracy to your results dictionary:

result_dic = {"epochs": []}

json_logging_callback = tf.keras.callbacks.LambdaCallback(
                on_epoch_begin=lambda epoch, logs: [learning_rate],
                on_epoch_end=lambda epoch, logs:
                result_dic["epochs"].append({
                    'epoch': epoch + 1, 
                    'acc': str(logs['accuracy']), 
                    'val_acc': str(logs['val_accuracy']),
                    'test_acc': str(logs['test_accuracy'])
                }))

And then just use both callbacks in your fit function but note the order of the callbacks:

model.fit(x_train, y_train,
                      validation_data=(x_val, y_val),
                      batch_size=batch_size,
                      epochs=epochs,
                      callbacks=callbacks=[TestCallback((x_test, y_test)), json_logging_callback])
user_96578610
Related