MeanAveragePrecision - bug in `max_detection_thresholds` about torchmetrics

[SkafteNicki]

Hi @nisyad-ms, thanks for reporting this issue.
Sadly this is due to a known bug in the official pycocotools backend, that we use for computations. Specifically, this line:
https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L460
should have been

stats[0] = _summarize(1, maxDets=self.params.maxDets[2])

for it to work. Sadly the repo is not really maintained anymore, but is still considered the official reference for mAP.

Instead you can install the faster-coco-eval backend (https://github.com/MiXaiLL76/faster_coco_eval) which we also supports. This backend have implemented the fix so your code calculates the correct value.

metric = MeanAveragePrecision(iou_type="bbox", max_detection_thresholds=[1, 10, 50], backend="faster_coco_eval")
metric.update(preds, target)
result = metric.compute()
print(result)

#{'map': tensor(1.), ...

Closing issue because we really cannot fix this on our side.

[nisyad-ms]

Thanks @SkafteNicki for the information. How to ensure the faster-coco-eval backend is used? Just installing it will do? Thanks again.

[SkafteNicki]

Thanks @SkafteNicki for the information. How to ensure the faster-coco-eval backend is used? Just installing it will do? Thanks again.

Sorry I should have specified that. You install the backend with pip install faster-coco-eval and then when initializing the MeanAveragePrecision class you need to set the backend argument to "faster_coco_eval".