What are the best metrics for Multi-Object Tracking (MOT) evaluation and why?

user_9738227 0
Viewed 628

I want to compare multiple computer vision Multi-Object Tracking (MOT) methods on my own dataset, so first I want to choose the best metrics for this task. I have carried out some research in scientific literature and I come to the conclusion that there are three main metrics sets:

  1. Metrics from "Tracking of Multiple, Partially Occluded Humans based on Static Body Part Detection"
  2. CLEAR MOT metrics
  3. ID scores

Therefore, I wonder to which of the above metrics should I attach the greatest importance?

And I would like to ask if anyone has encountered a similar issue and has any thoughts on this topic that could justify and help me to choose the best metrics for the above task.

object-detection deep-learning tracking computer-vision object
3 Answers

You can refer to the metrics used in the MOT Challenge.

Here's the results for the MOT 2020 Challenge and they have included the metrics used here: https://motchallenge.net/results/MOT20/

Based on the MOT 20 paper, they said at section 4.1.7 (page 7):

As we have seen in this section, there are a number of reasonable performance measures to assess the quality of a tracking system, which makes it rather difficult to reduce the evaluation to one single number. To nevertheless give an intuition on how each tracker performs compared to its competitors, we compute and show the average rank for each one by ranking all trackers according to each metric and then averaging across all performance measures.

user_144612620

the metrics that you choose related to what your goal after multiple object tracking like : if your goals interest tracking the people inside scence you should the metric ID switch very low and so on ...

you should find the metrics related to your goals.

user_144774870

I know this is old but I see nobody mentioning HOTA (https://arxiv.org/pdf/2009.07736.pdf). This has become the new standard for multi-object tracking as can be seen here: https://arxiv.org/abs/2202.13514 and https://arxiv.org/pdf/2110.06864.pdf

MOTA and IDF1 overemphasize detection and association respectively. HOTA explicitly measures both types of errors and combines these in a balanced way. HOTA also incorporates measuring the localization accuracy of tracking results which isn’t present in either MOTA or IDF1.

user_95067730
Related