using sklearn macro f1-score as a metric in tensorflow.keras

Viewed 2299

I have defined custom metric for tensorflow.keras to compute macro-f1-score after every epoch as follows:

from tensorflow import argmax as tf_argmax
from sklearn.metric import f1_score

def macro_f1(y_true, y_pred):
    # labels are one-hot encoded. so, need to convert
    # [1,0,0] to 0 and
    # [0,1,0] to 1 and
    # [0,0,1] to 2. Then pass these arrays to sklearn f1_score.
    y_true = tf_argmax(y_true, axis=1)
    y_pred = tf_argmax(y_pred, axis=1)
    return f1_score(y_true, y_pred, average='macro')

and using it during model compilation

model_4.compile(loss = 'categorical_crossentropy',
                optimizer = Adam(lr=init_lr, decay=init_lr / num_epochs),
                metrics = [Recall(name='recall') #, weighted_f1
                           macro_f1])

and when i try to fit like this:

history_model_4 = model_4.fit(train_image_generator.flow(x=train_imgs, y=train_targets, batch_size=batch_size),
                            validation_data = (val_imgs, val_targets),
                            epochs=num_epochs,
                            class_weight=mask_weights_train,
                            callbacks=[model_save_cb, early_stop_cb, epoch_times_cb],
                            verbose=2)

this is the error:

OperatorNotAllowedInGraphError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    <ipython-input-57-a890ea61878e>:6 macro_f1  *
        return f1_score(y_true, y_pred, average='macro')
    /usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1095 f1_score  *
        return fbeta_score(y_true, y_pred, 1, labels=labels,
    /usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1217 fbeta_score  *
        _, _, f, _ = precision_recall_fscore_support(y_true, y_pred,
    /usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1478 precision_recall_fscore_support  *
        labels = _check_set_wise_labels(y_true, y_pred, average, labels,
    /usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1301 _check_set_wise_labels  *
        y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    /usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:80 _check_targets  *
        check_consistent_length(y_true, y_pred)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:209 check_consistent_length  *
        uniques = np.unique(lengths)
    <__array_function__ internals>:6 unique  **
        
    /usr/local/lib/python3.6/dist-packages/numpy/lib/arraysetops.py:263 unique
        ret = _unique1d(ar, return_index, return_inverse, return_counts)
    /usr/local/lib/python3.6/dist-packages/numpy/lib/arraysetops.py:311 _unique1d
        ar.sort()
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:877 __bool__
        self._disallow_bool_casting()
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:487 _disallow_bool_casting
        "using a `tf.Tensor` as a Python `bool`")
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:474 _disallow_when_autograph_enabled
        " indicate you are trying to use an unsupported feature.".format(task))

    OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

What caused such errors and how do I fix it and use it as one of my evaluation metrics at the end of ever y epoch?

EDIT 1:
note: all of this has been done in a jupyter notebook, i have added ">>>"s to seperate lines

# getting a batch to pass to model
>>> a_batch = train_image_generator.flow(x=train_imgs, y=train_targets, batch_size=batch_size).next()
# checking its' type to ensure that it's what i though it is
>>> type(a_batch)
# passing the batch to the model
>>> logits = model_4(a_batch)
# checking the type of output
>>> type(logits)
tensorflow.python.framework.ops.EagerTensor
# extracting only the passed targets to calculate f1-score
>>> _, dummy_targets = a_batch
# checking it's type
>>> type(dummy_targets)
numpy.ndarray
>>> macro_f1(y_true=dummy_targets, y_pred=logits)
0.0811965811965812
1 Answers

sklearn is not TensorFlow code - it is always recommended to avoid using arbitrary Python code in TF that gets executed inside TF's execution graph.

TensorFlow addons already has an implementation of the F1 score (tfa.metrics.F1Score), so change your code to use that instead of your custom metric

Make sure you pip install tensorflow-addons first and then

import tensorflow_addons as tfa

model_4.compile(loss = 'categorical_crossentropy',
                optimizer = Adam(lr=init_lr, decay=init_lr / num_epochs),
                metrics = [Recall(name='recall') #, weighted_f1
                           tfa.metrics.F1Score(average='macro')])
Related