TY - JOUR T1 - Online Empirical Evaluation of Tracking Algorithms JF - IEEE Transactions on Pattern Analysis and Machine Intelligence Y1 - 2010 A1 - Wu,Hao A1 - Sankaranarayanan,A. C A1 - Chellapa, Rama KW - Back KW - Biomedical imaging KW - Computer vision KW - Filtering KW - formal model validation techniques KW - formal verification KW - ground truth KW - Kanade Lucas Tomasi feature tracker KW - Karhunen-Loeve transforms KW - lighting KW - Markov processes KW - mean shift tracker KW - model validation. KW - online empirical evaluation KW - particle filtering (numerical methods) KW - Particle filters KW - Particle tracking KW - performance evaluation KW - receiver operating characteristic curves KW - Robustness KW - SNR KW - Statistics KW - Surveillance KW - time reversed Markov chain KW - tracking KW - tracking algorithms KW - visual tracking AB - Evaluation of tracking algorithms in the absence of ground truth is a challenging problem. There exist a variety of approaches for this problem, ranging from formal model validation techniques to heuristics that look for mismatches between track properties and the observed data. However, few of these methods scale up to the task of visual tracking, where the models are usually nonlinear and complex and typically lie in a high-dimensional space. Further, scenarios that cause track failures and/or poor tracking performance are also quite diverse for the visual tracking problem. In this paper, we propose an online performance evaluation strategy for tracking systems based on particle filters using a time-reversed Markov chain. The key intuition of our proposed methodology relies on the time-reversible nature of physical motion exhibited by most objects, which in turn should be possessed by a good tracker. In the presence of tracking failures due to occlusion, low SNR, or modeling errors, this reversible nature of the tracker is violated. We use this property for detection of track failures. To evaluate the performance of the tracker at time instant t, we use the posterior of the tracking algorithm to initialize a time-reversed Markov chain. We compute the posterior density of track parameters at the starting time t = 0 by filtering back in time to the initial time instant. The distance between the posterior density of the time-reversed chain (at t = 0) and the prior density used to initialize the tracking algorithm forms the decision statistic for evaluation. It is observed that when the data are generated by the underlying models, the decision statistic takes a low value. We provide a thorough experimental analysis of the evaluation methodology. Specifically, we demonstrate the effectiveness of our approach for tackling common challenges such as occlusion, pose, and illumination changes and provide the Receiver Operating Characteristic (ROC) curves. Finally, we also s how the applicability of the core ideas of the paper to other tracking algorithms such as the Kanade-Lucas-Tomasi (KLT) feature tracker and the mean-shift tracker. VL - 32 SN - 0162-8828 CP - 8 M3 - 10.1109/TPAMI.2009.135 ER -