Mini project 3 · CMU 16-720
Lucas-Kanade tracking.
Tracking by estimating optical flow, in three escalating forms: following a template through a scene, separating moving objects from a moving camera, and making the whole thing efficient enough to be practical.
Three trackers
I implemented the Lucas-Kanade method, a widely used approach for tracking by estimating optical flow. The first tracker follows a moving template in the scene, implemented as 2D tracking with a pure-translation warp, including a correction for the template-drift problem: because each frame's template is updated from the last, small alignment errors compound until the tracker slides off its target entirely.
The second estimates the dominant affine motion across a sequence of images and then identifies the pixels that do not fit it. That inversion is the clever part: rather than detecting moving objects directly, you model the motion of the background and treat whatever disagrees as the moving object. It was tested on aerial footage of moving ants and of vehicles seen from a non-stationary camera.
Finally, I added the inverse-compositional extension of Lucas-Kanade, which restructures the optimization so that the expensive Hessian computation can be done once up front rather than at every iteration, making the tracking substantially more efficient.