Consider the following object detection dataset in coco format (let's call it gt)
{"info": {}, "license": {},
"images": [{"id": 1, "width": 640, "height": 480, "file_name": "xyz.jpeg", "data_captured": null}],
"annotations": [{
"id": 1, "image_id": 1, "category_id": 2,
"bbox": [0.2, 0.2, 0.5, 0.2],
"area": 0.01125, "iscrowd": 0
}],
"categories": [
{"id": 1, "name": "category_1"},
{"id": 2, "name": "category_2"}
]}
And predictions also in coco format
[{
"image_id": 1, "category_id": 2,
"bbox": [0.2, 0.2, 0.5, 0.2],
"score": 0.99
}]
Evaluation on these files using pycocotools correctly outputs 100% mAP@IoU0.5 and other metrics.
If I replace gt['annotations'][0]['id'] to 0 like this
{"info": {}, "license": {},
"images": [{"id": 1, "width": 640, "height": 480, "file_name": "xyz.jpeg", "data_captured": null}],
"annotations": [{
"id": 0, "image_id": 1, "category_id": 2, # replace id here
"bbox": [0.2, 0.2, 0.5, 0.2],
"area": 0.01125, "iscrowd": 0
}],
"categories": [
{"id": 1, "name": "category_1"},
{"id": 2, "name": "category_2"}
]}
The evaluation returns 0 mAP@IoU0.5. Question:
Is this bug in the pycocotools library or is there some special meaning to id=0 (similar to how category_id 0 is sometimes reserved for background?)
Minimal code sample to run an evaluation
!pip install pycocotools
import json
from pycocotools.cocoeval import COCOeval
from pycocotools.coco import COCO
gt_path = './gt.json'
labels_path = './labels.json'
coco_gt = COCO(gt_path)
anns = json.load(open(labels_path))
coco_dt = coco_gt.loadRes(labels_path)
coco_evaluator = COCOeval(coco_gt, coco_dt, 'bbox')
coco_evaluator.evaluate()
coco_evaluator.accumulate()
coco_evaluator.summarize()