Large Displacement Optical Flow Matlab Code
[Download]
The code implements a coarsetofine variational framework for optical flow estimation that incorporates descriptor matches in addition to brightness and gradient constrancy constraints. Descriptor matches help flow estimation under large displacements of small structures, missed in the coarsetofine pyramid. The code is based on [1] and [2]. We denote with w(x)=(u,v) the displacement field between two images I_1 and I_2. Following [2], we obtain descriptor matches w_descr(x) by matching densely sampled HOG descriptors in the two images via approximate nearest neighbor search. We then compute w(x) by minimizing the energy functional:
In the above equation,
the first term encodes intensity and gradient constancy, the second term encodes robust smoothness constraint and
the last term biases the displacement field w(x) towards the confident descriptor matches w_descr(x). \delta(x) is a delta function indicating if a descriptor match is available in location and \rho(x) denotes confidence of the match.
Descriptor matches are obtained by matching densely sampled HOG descriptors in the two images.
We minimize the above functional in a coarsetofine scheme: at each pyramid level the nonlinear Eulerlagrange equations are solved via fixpoint iterations. At each iteration, the sparse linear system is solved
via succesive overelaxation (SOR) as described in [1]. Intuitively, at each pyramid level, the pixel displacement estimates and pairwise/unary pixel potentials are updated iteratively: given the current
pixel motion estimates w(x), the unary and pairwise terms are updated and so on and so forth. Once a fixpoint has been reached, the flow field is upsampled to initialize the computation at a finer pyramid level.
Examples
The (back) warped second image should coincide with the first except for places that get occluded or disoccluded. In those places optical flow is not well defined and may be arbitrary. Forwardbackward consistency check can discard flow vectors in occluded regions.
 

How to use?
Download the code folder and run start_up.m while being inside the folder, to compile the mex files. Then you should be able to run the demo. We observed that the image dowsampling factor for creating the image pyramid needs to be close to one for the linearizations to be valid and the small displacement assumption to hold at each pyramid level, to deal effectively with very large displacements. It can be varied according to the ammount of displacement, small downsampling factor results in fewer pyramid levels and thus smaller running time. Please report comments/bugs to katef@seas.upenn.edu.References
[1]  T. Brox, A. Bruhn, N. Papenberg, and J.Weickert. High accuracy optical flow based on a theory for warping. In European Conference on Computer Vision (ECCV), pages 25–36, 2004. 
[2]  T. Brox, J. Malik. Large Displacement Optical Flow: descriptor matching in variational motion estimation. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(3): 500513, 2011 
Log
Dec. 1st 2013: Bug corrected in line 109 of variational_descriptor_flow.mLast update: Dec, 2013.