Class wise precision and recall for multi class classification in Tensorflow?

Here's a solution that is working for me for a problem with n=6 classes. If you have many more classes this solution is probably slow and you should use some sort of mapping instead of a loop.

Assume you have one hot encoded class labels in rows of tensor labels and logits (or posteriors) in tensor labels. Then, if n is the number of classes, try this:

y_true = tf.argmax(labels, 1)
y_pred = tf.argmax(logits, 1)

recall = [0] * n
update_op_rec = [[]] * n

for k in range(n):
    recall[k], update_op_rec[k] = tf.metrics.recall(
        labels=tf.equal(y_true, k),
        predictions=tf.equal(y_pred, k)
    )

Note that inside tf.metrics.recall, the variables labels and predictions are set to boolean vectors like in the 2 variable case, which allows the use of the function.

2 facts:

1. As stated in other answers, Tensorflow built-in metrics precision and recall don't support multi-class (the doc says will be cast to bool)

2. There are ways of getting one-versus-all scores by using precision_at_k by specifying the class_id, or by simply casting your labels and predictions to tf.bool in the right way.

Because this is unsatisfying and incomplete, I wrote tf_metrics, a simple package for multi-class metrics that you can find on github. It supports multiple averaging methods like scikit-learn.

Example

import tensorflow as tf
import tf_metrics

y_true = [0, 1, 0, 0, 0, 2, 3, 0, 0, 1]
y_pred = [0, 1, 0, 0, 1, 2, 0, 3, 3, 1]
pos_indices = [1]        # Metrics for class 1 -- or
pos_indices = [1, 2, 3]  # Average metrics, 0 is the 'negative' class
num_classes = 4
average = 'micro'

# Tuple of (value, update_op)
precision = tf_metrics.precision(
    y_true, y_pred, num_classes, pos_indices, average=average)
recall = tf_metrics.recall(
    y_true, y_pred, num_classes, pos_indices, average=average)
f2 = tf_metrics.fbeta(
    y_true, y_pred, num_classes, pos_indices, average=average, beta=2)
f1 = tf_metrics.f1(
    y_true, y_pred, num_classes, pos_indices, average=average)

I believe you cannot do multiclass precision, recall, f1 with the tf.metrics.precision/recall functions. You can use sklearn like this for a 3 class scenario:

from sklearn.metrics import precision_recall_fscore_support as score

prediction = [1,2,3,2] 
y_original = [1,2,3,3]

precision, recall, f1, _ = score(y_original, prediction)

print('precision: {}'.format(precision))
print('recall: {}'.format(recall))
print('fscore: {}'.format(f1))

This will print an array of precision, recall values but format it as you like.

I have been puzzled by this problem for quite a long time. I know this problem can be solved by sklearn, but I really want to solve this by Tensorflow's API. And by reading its code, I finally figure out how this API works.

tf.metrics.precision_at_k(labels, predictions, k, class_id)

• Firstly, let's assume this is a 4 classes problem.
• Secondly, we have two samples which their labels are 3 and 1 and their predictions are [0.5,0.3,0.1,0.1], [0.5,0.3,0.1,0.1] .According to our predictions, we can get the result that the two samples has been predicted as 1,1.
• Thirdly, if you want to get the precision of class 1, use the formula TP/(TP+FP), and we assume the result is 1/(1+1)=0.5. Because the two samples both have been predicted as 1, but one of the them is actually 3, so the TP is 1, the FP is 1, and the result is 0.5.

• Finally, let's use this API to verify our assumption.

  import tensorflow as tf
  
  labels = tf.constant([[2],[0]],tf.int64)
  predictions = tf.constant([[0.5,0.3,0.1,0.1],[0.5,0.3,0.1,0.1]])
  
  metric = tf.metrics.precision_at_k(labels, predictions, 1, class_id=0)
  
  sess = tf.Session()
  sess.run(tf.local_variables_initializer())
  
  precision, update = sess.run(metric)
  print(precision) # 0.5
  

NOTICE

• k isn't the number of classes. It represents the number of what we want to sort, which means the last dimension of predictions must match the value of k.

• class_id represents the Class for which we want binary metrics.

• If k=1, means that we won't sort the predictions, because what we want to do is actually a binary classificaion, but referring to different classes. So if we sort the predictions, the class_id will be confused and the result will be wrong.

• And one more important thing is that if we want to get the right result, the input of label should minus 1 because the class_id actually represents the index of the label, and the subscript of label starts with 0.

There is a way to do this in TensorFlow.

tf.metrics.precision_at_k(labels, predictions, k, class_id)

set k = 1 and set corresponding class_id. For example class_id=0 to calculate the precision of first class.

I believe TF does not provide such functionality yet. As per the docs (https://www.tensorflow.org/api_docs/python/tf/metrics/precision), it says both the labels and predictions will be cast to bool, and so it relates only to binary classification. Perhaps it's possible to one-hot encode the examples and it would work? But not sure about this.

Here's a complete example from predicting in Tensorflow to reporting via scikit-learn:

import tensorflow as tf
from sklearn.metrics import classification_report

# given trained model `model` and test vector `X_test` gives `y_test`
# where `y_test` and `y_predicted` are integers, who labels are indexed in 
# `labels`
y_predicted = tf.argmax(model.predict(X_test), axis=1)

# Confusion matrix
cf = tf.math.confusion_matrix(y_test, y_predicted)
plt.matshow(cf, cmap='magma')
plt.colorbar()
plt.xticks(np.arange(len(labels)), labels=labels, rotation=90)
plt.yticks(np.arange(len(labels)), labels=labels)
plt.clim(0, None)

# Report
print(classification_report(y_test, y_predicted, target_names=labels))