Calculating YOLO mAP against test dataset

Question

AFAIK YOLO calculates mAP against validation dataset during training. Now is it possible to calculate the same against unseen test dataset ?

Command:

./darknet detector map obj.data yolo-obj.cfg yolo-obj_best.weights

obj.data:

classes = 1
train = train.txt
valid = test.txt
names = classes.txt
backup = backup

I have directed valid to test dataset containing annotated images. But I always get the following result:

 calculation mAP (mean average precision)...
44
 detections_count = 50, unique_truth_count = 43  
class_id = 0, name = traffic_light, ap = 100.00%     (TP = 43, FP = 0) 

 for conf_thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00 
 for conf_thresh = 0.25, TP = 43, FP = 0, FN = 0, average IoU = 85.24 % 

 IoU threshold = 50 %, used Area-Under-Curve for each unique Recall 
 mean average precision (mAP@0.50) = 1.000000, or 100.00 % 
Total Detection Time: 118 Seconds

It's not that I'm not happy with 100% mAP, but it's definitely wrong isn't it?

Any advice would be greatly appreciated.

Regards,

Setnug

user_6632342 · Answer

Now is it possible to calculate the same against unseen test dataset ?

Yes, mAP calculation needs images with corresponding labels/annotation that's all.

I have directed valid to test dataset containing annotated images.

Yes, this is the way to do what you wanted.

There is a possibility that what you're seeing here is this known bug, provided you're using old code and haven't updated after that. In that case suggest you to pull the latest darknet and try.

Note that if the model is trained really well and if your test set is simple in terms of complexity (though it's unseen) or it's visually similar to that of train set, it's possible to get such numbers as well, as we're talking about small number of test samples.

Calculating YOLO mAP against test dataset

1 Answers