The idea is to set up TensorBoard with good metrics to track my model training and evaluate model performance. However, most of the metrics seem incorrect (especially precision/recall/false_negative_count) I am using tf v1.8.0.
To create the metrics, I use the following peace of code:
def as_keras_metric(method, **kwargs):
@functools.wraps(method)
def wrapper(self, args):
""" Wrapper for turning tensorflow metrics into keras metrics """
value, update_op = method(self, args, **kwargs)
tf.keras.backend.get_session().run(tf.local_variables_initializer())
with tf.control_dependencies([update_op]):
value = tf.identity(value)
return value
return wrapper
def create_metrics():
auc_roc = as_keras_metric(tf.metrics.auc)
recall = as_keras_metric(tf.metrics.recall)
precision = as_keras_metric(tf.metrics.precision)
fn = as_keras_metric(tf.metrics.false_negatives)
fp = as_keras_metric(tf.metrics.false_positives)
kwargs = {'specificity': 0.95}
sensitivity_at_specificity = as_keras_metric(tf.metrics.sensitivity_at_specificity, **kwargs)
mean_per_class_accuracy = as_keras_metric(tf.metrics.mean_per_class_accuracy, **{'num_classes': 2})
return [fn, fp, auc_roc, recall, precision, sensitivity_at_specificity, mean_per_class_accuracy, tf.keras.metrics.binary_accuracy]
I compile the model using:
model.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=create_metrics())
and fit using
model.fit_generator(generator=KerasSequence)
Now here is the problem:
My data is imbalanced, I have around 2% of observations belonging to the +ve class. Let's look at the last few epochs.
Epoch 16/50 6/7 [========================>.....] - ETA: 0s - loss: 5.9874 - auc: 0.9171 - recall: 1.0000 - precision: 0.0068 - sensitivity_at_specificity: 0.8050 - mean_per_class_accuracy: 0.5000 - false_negatives: 0.0000e+00 - false_positives: 13570.6667 - binary_accuracy: 1.0000
Epoch 50/50 6/7 [========================>.....] - ETA: 0s - loss: 0.0205 - auc: 0.9207 - recall: 1.0000 - precision: 0.0069 - sensitivity_at_specificity: 0.8494 - mean_per_class_accuracy: 0.5000 - false_negatives: 0.0000e+00 - false_positives: 43821.6667 - binary_accuracy: 1.0000
I know that 6 out of 500 samples were predicted as +ve class. So I don't understand how it's possible to have these metrics. They are mutually exclusive. Recall at 1 and precision so low. Which suggests that I actually predict 1 for all observations, when in reality only 5/500 are predicted to be +ve class. Additionaly, how is it possible to get false_positive count at 43000... when I only have 500 training samples. And it's growing with every epoch
Screenshots
The loss goes to 0 (overfitting), but the false positive count increases per epoch... why? Interestingly, after the 1st epoch, false positive count is at 200. This makes me thing that the metrics don't reset after epoch end and keep adding.
