THE METRICS FOR TRACKING ALGORITHMS EVALUATION
Abstract
The work is devoted to a review of existing metrics for assessing the quality of the task of tracking objects on video with various algorithms. When evaluating tracking algorithms for their subsequent com-parison, it is not enough to use one metric, and algorithms should be evaluated using a set of different independent estimates. To this end, a study was conducted of existing metrics for evaluating algorithms, the results of which are given in the article. The review involves many different approaches to evaluating algorithms. For example, approaches based on the assessment of the definition of the center of the track-ing object, which are one of the first and still popular metrics for evaluating tracking algorithms. The main disadvantages of such approaches include the difficulty of determining the true center of the object, as well as the interpretation of estimates for various sizes of the object. To eliminate these shortcomings, anew metric is introduced in the article: an unbiased (window) error in determining the center of an object, which takes into account the constant component of the error in determining the center. Other approach-es include metrics based on the analysis of the intersection over union. Also, the article considers ap-proaches based on the analysis of tracking failures, which take into account the tracking length and fail-ure rate. A new method is proposed for evaluating algorithms in case of loss of visual contact with an tracking object, taking into account the number of frames in which visual contact with the object was lost. During the study, approaches to evaluating algorithms for simultaneous tracking of several objects were considered. Integral metrics were proposed whose task is to obtain a comprehensive assessment of the tracking algorithm. For the formation of a comprehensive assessment, it is desirable to use various un-correlated metrics. Complex estimates provide the ability to compare algorithms with each other. As a comprehensive assessment, the article proposes the use of a metric combining the accuracy and robust-ness of the algorithm. As a rule, the intersection over union is used as the accuracy metric, however, for problems where the accuracy of tracking the center of the object is fundamental, the authors propose using an unbiased error in determining the center as the accuracy metric.
References
2. Godec M., Roth P., Bischof H. Hough-based tracking of nonrigid objects, ICCV. Barcelona: IEEE, 2011, pp. 81-88.
3. Wang D., Lu H., Yang M. Online object tracking with sparse prototypes, IEEE Image Procesing Conference, 2013, Vol. 22, No. 1, pp. 314-325.
4. Wu Y., Lim J., Yang M.-h. Online Object Tracking: A Benchmark, CVPR, 2013.
5. Everingham M., Gool L., Williams I., Winn J., Zisserman A. The Pascal Visual Object Classes (VOC) Challenge, IJCV, 2009, Vol. 88, No. 2, pp. 303-338.
6. Kristan M., Kovaˇciˇc S., Leonardis A., Pers J. A two-stage dynamic model for visual tracking, Transactions Systems Man Cybernetic Part B, 2010, Vol. 40, No. 6, pp. 1505-1520.
7. Khan Z., Balch T., Dellaert F MCMC-based particle filtering for tracking a variable number of interacting targets, IEEE Transactions Pattern Analysis, 2005, Vol. 27, pp. 1805-1819.
8. Kasturi R., Goldgof D., Soundararajan P., Manohar V., Garofolo J., Bowers R., Boonstra M., Korzhova V., Zhang J. Framework for performance evaluation of face, text, and vehicle detection and tracking in video: data, metrics, and protocol, TPAMI, 2009, Vol. 31, No. 2, pp. 319-326.
9. Ventsel' E.S., Ovcharov L.A. Teoriya veroyatnosti i ee inzhenernye prilozheniya: ucheb. posobie dlya vuzov [Probability theory and its engineering applications: a textbook for univer-sities]. 2nd ed. Moscow: Vyssh. shk., 2000, 480 p. 10. Yang M., Wu Y., Hua G. Context–aware visual tracking, IEEE transactions on pattern analysis and machine intelligence, 2006, No. 31, pp. 1195-1209. 11. Stauffer C., Grimson W. Learning patterns of activity using real–time tracking // IEEE Trans-actions on Pattern Analysis and Machine Intelligence. – 2000. – P. 747-757. 12. Sundaresan A., Chellappa R. Multi–camera tracking of articulated human motion using shape and motion cues, IEEE Transactions on Image Processing, 2009, pp. 2114-2126. 13. Smeulders A. M., Chu D., Cucchiara, S. Calderara, A., Shah M. Visual Tracking: an Experi-mental Survey, TPAMI, 2013, Vol. 21, No. 3, pp. 152-168. 14. Henriques F., Caseiro R., Martins P., Batista J. High-Speed Tracking with Kernelized Corre-lation Filters, TPAMI, 2014, Vol. 42, No. 5, pp. 345-362.
15. Frey J. Dueck D. Clustering by Passing Messages Between Data Points, Science Today, 2007, Vol. 315, pp. 972-976.
16. Smith K., Gatica-Perez D., Odobez J. Evaluating Multi-Object Tracking, CVPR Work, 2005, Vol. 3, IEEE, pp. 32-36.
17. Black J., Ellis T., Rosin P. A novel method for video tracking performance evaluation, VS-PETS, 2003, pp. 125-132.
18. Kao E., Daggett M., Hurley M. An information theoretic approach for tracker performance evaluation, CVPR, 2009, pp. 1523-1529.
19. Bashir F. Porikli F. Performance Evaluation of Object Detection and Tracking Systems, PETS, 2006, pp. 190-203. 20. Checka N., Wilson K., Rangarajan V., Darrell T. A probabilistic framework for multi-modal multi-person tracking, Proceedings of the IEEE Workshop on Multi-Object Tracking (WOMOT ’03), 2003, pp. 203-212.
20. Checka N., Wilson K., Rangarajan V., Darrell T. A probabilistic framework for multi-modal multi-person tracking, Proceedings of the IEEE Workshop on Multi-Object Tracking (WOMOT ’03), 2003, pp. 203-212.
