tensorflow checkpoint messageno match with training metric

user_3330229 0
Viewed 47

I set up checkpoint callback with checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath=ckpt_path, save_best_only=True, monitor='val_auc', verbose=1). When I take look at my training log, it seems mismatching.

：Epoch 00004: val_auc improved from 0.96440 to 0.96298, saving model to xxxxxxx
py log：2878/2878 - 352s - loss: 0.2071 - tp: 1207371.0000 - fp: 66819.0000 - tn: 1484009.0000 - fn: 187884.0000 - accuracy: 0.9135 - precision: 0.9476 - recall: 0.8653 - auc: 0.9698 - pr: 0.9731 - val_loss: 0.2388 - val_tp: 338551.0000 - val_fp: 5482.0000 - val_tn: 76038.0000 - val_fn: 49446.0000 - val_accuracy: 0.8830 - val_precision: 0.9841 - val_recall: 0.8726 - val_auc: 0.9630 - val_pr: 0.9921

Why val_auc improved from a bigger one to smaller one, and why it's 0.96298 not 0.9630 in the line below.

tf.keras keras tensorflow python
1 Answers

You need to set the mode parameter to max to save the max val_auc everytime.

checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath=ckpt_path, mode="max", save_best_only=True, monitor='val_auc', verbose=1)

The documentation for the parameter mode explains this:

mode: one of {'auto', 'min', 'max'}. If save_best_only=True, the decision to overwrite the current save file is made based on either the maximization or the minimization of the monitored quantity. For val_acc, this should be max, for val_loss this should be min, etc. In auto mode, the mode is set to max if the quantities monitored are 'acc' or start with 'fmeasure' and are set to min for the rest of the quantities.

Since the default value is auto and you are monitoring val_auc not acc or anything that starts with fmeasure it is set to min.

The val_auc in the logs are rounded to 4 decimal places that's why you see it as 0.9630 instead of 0.96298.

user_133376350
Related