Confidence thresholds on mean average precision calculation

user_15333912 0
Viewed 209

is there any rules for PR curve threshold because in sklearn.metrics.average_precision they automatically make threshold from the prob/confidence which can result in weird result if I have inputs like this:

y_true = np.array([0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
y_scores = np.array([ 0.7088982, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

it will output mAP = 0.93333. Sklearn implementation got that number because it automatically uses [0.7088982, 0] as the thresholds. When the prob threshold is 0 all of zero score will counted as positive resulting in high map. Is this a correct behavior ?

scikit-learn
1 Answers

A couple of considerations on your example:

  • the peculiarity of your y_scores, having two distinct values only, defines the length of your threshold. As you might see from source code and as you may logically imply, threshold is defined by the number of distinct values in y_scores.
  • then, your argument is correct and implicit in what the threshold represents. Actually, if the score is greater or equal than the threshold, the instance is assigned to the positive class. Therefore, in the case score=threshold=0 you'll have true positives only based on your y_true (and in turn the average precision is a weighted mean of precisions achieved at each threshold). Have a look also here to observe that

Precision values such that element i is the precision of predictions with score >= thresholds[i]

and

Decreasing recall values such that element i is the recall of predictions with score >= thresholds[i]

I'd also suggest you to have a look here to get a glimpse of how precision, recall and threshold are computed within precision_recall_curve().

user_133503410
Related