Large Displacement Optical Flow Matlab Code

[Download]

The code implements a coarse-to-fine variational framework for optical flow estimation that incorporates descriptor matches in addition to brightness and gradient constrancy constraints. Descriptor matches help flow estimation under large displacements of small structures, missed in the coarse-to-fine pyramid. The code is based on [1] and [2]. We denote with w(x)=(u,v) the displacement field between two images I_1 and I_2. Following [2], we obtain descriptor matches w_descr(x) by matching densely sampled HOG descriptors in the two images via approximate nearest neighbor search. We then compute w(x) by minimizing the energy functional:

In the above equation, the first term encodes intensity and gradient constancy, the second term encodes robust smoothness constraint and the last term biases the displacement field w(x) towards the confident descriptor matches w_descr(x). \delta(x) is a delta function indicating if a descriptor match is available in location and \rho(x) denotes confidence of the match. Descriptor matches are obtained by matching densely sampled HOG descriptors in the two images.
We minimize the above functional in a coarse-to-fine scheme: at each pyramid level the non-linear Euler-lagrange equations are solved via fix-point iterations. At each iteration, the sparse linear system is solved via succesive overelaxation (SOR) as described in [1]. Intuitively, at each pyramid level, the pixel displacement estimates and pairwise/unary pixel potentials are updated iteratively: given the current pixel motion estimates w(x), the unary and pairwise terms are updated and so on and so forth. Once a fix-point has been reached, the flow field is upsampled to initialize the computation at a finer pyramid level.

Examples

The (back) warped second image should coincide with the first except for places that get occluded or dis-occluded. In those places optical flow is not well defined and may be arbitrary. Forward-backward consistency check can discard flow vectors in occluded regions.
RGB

Flow field

descriptor matching
t
t+1
image 1 & (back)warped image2

RGB

Flow field

descriptor matching
t
t+1
image 1 & (back)warped image2

How to use?

Download the code folder and run start_up.m while being inside the folder, to compile the mex files. Then you should be able to run the demo. We observed that the image dowsampling factor for creating the image pyramid needs to be close to one for the linearizations to be valid and the small displacement assumption to hold at each pyramid level, to deal effectively with very large displacements. It can be varied according to the ammount of displacement, small downsampling factor results in fewer pyramid levels and thus smaller running time. Please report comments/bugs to katef@seas.upenn.edu.

References

[1] T. Brox, A. Bruhn, N. Papenberg, and J.Weickert. High accuracy optical flow based on a theory for warping. In European Conference on Computer Vision (ECCV), pages 25–36, 2004.
[2] T. Brox, J. Malik. Large Displacement Optical Flow: descriptor matching in variational motion estimation. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(3): 500-513, 2011

Log

Dec. 1st 2013: Bug corrected in line 109 of variational_descriptor_flow.m
Last update: Dec, 2013.