Keras TensorBoard showing incorrect Recall/Precision metrics

Viewed 540

The idea is to set up TensorBoard with good metrics to track my model training and evaluate model performance. However, most of the metrics seem incorrect (especially precision/recall/false_negative_count) I am using tf v1.8.0.

To create the metrics, I use the following peace of code:

def as_keras_metric(method, **kwargs):
    @functools.wraps(method)
    def wrapper(self, args):
        """ Wrapper for turning tensorflow metrics into keras metrics """
        value, update_op = method(self, args, **kwargs)
        tf.keras.backend.get_session().run(tf.local_variables_initializer())
        with tf.control_dependencies([update_op]):
            value = tf.identity(value)
        return value
    return wrapper


def create_metrics():
    auc_roc = as_keras_metric(tf.metrics.auc)
    recall = as_keras_metric(tf.metrics.recall)
    precision = as_keras_metric(tf.metrics.precision)

    fn = as_keras_metric(tf.metrics.false_negatives)
    fp = as_keras_metric(tf.metrics.false_positives)

    kwargs = {'specificity': 0.95}
    sensitivity_at_specificity = as_keras_metric(tf.metrics.sensitivity_at_specificity, **kwargs)

    mean_per_class_accuracy = as_keras_metric(tf.metrics.mean_per_class_accuracy, **{'num_classes': 2})

    return [fn, fp, auc_roc, recall, precision, sensitivity_at_specificity, mean_per_class_accuracy, tf.keras.metrics.binary_accuracy]

I compile the model using:

model.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=create_metrics())

and fit using

model.fit_generator(generator=KerasSequence)

Now here is the problem:

My data is imbalanced, I have around 2% of observations belonging to the +ve class. Let's look at the last few epochs.

Epoch 16/50 6/7 [========================>.....] - ETA: 0s - loss: 5.9874 - auc: 0.9171 - recall: 1.0000 - precision: 0.0068 - sensitivity_at_specificity: 0.8050 - mean_per_class_accuracy: 0.5000 - false_negatives: 0.0000e+00 - false_positives: 13570.6667 - binary_accuracy: 1.0000

Epoch 50/50 6/7 [========================>.....] - ETA: 0s - loss: 0.0205 - auc: 0.9207 - recall: 1.0000 - precision: 0.0069 - sensitivity_at_specificity: 0.8494 - mean_per_class_accuracy: 0.5000 - false_negatives: 0.0000e+00 - false_positives: 43821.6667 - binary_accuracy: 1.0000

I know that 6 out of 500 samples were predicted as +ve class. So I don't understand how it's possible to have these metrics. They are mutually exclusive. Recall at 1 and precision so low. Which suggests that I actually predict 1 for all observations, when in reality only 5/500 are predicted to be +ve class. Additionaly, how is it possible to get false_positive count at 43000... when I only have 500 training samples. And it's growing with every epoch

Screenshots

enter image description here

The loss goes to 0 (overfitting), but the false positive count increases per epoch... why? Interestingly, after the 1st epoch, false positive count is at 200. This makes me thing that the metrics don't reset after epoch end and keep adding.

0 Answers
Related