Home
Uni-Logo
 

Video Segmentation

A major research focus in our group is video segmentation. One approach to video segmentation is by means of motion segmentation. Motion segmentation allows to retrieve object regions in a fully unsupervised manner. Another task is to propagate a segmentation given in one frame to the whole video. In 2014 and 2016, we co-organized a Workshop on video segmentation.



Thomas Brox, J. Malik
European Conference on Computer Vision (ECCV), Springer, LNCS, Sept. 2010

We found that the long term analysis of point trajectories rather than just two-frame optical flow is very powerful. The method can extract moving objects in a fully unsupervised manner from video shots.

Peter Ochs, J. Malik, Thomas Brox
IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(6):1187-1200, 2014

We proposed an overall motion segmentation system that combines above clustering of point trajectories with a dense segmentation. We also introduced the Freiburg-Berkeley motion segmentation benchmark, a dataset with 59 videos and pixel-accurate ground truth segmentation annotation.

IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2012

We improved the previous technique by adding the capability to deal with scaling and rotational motion. This leads to spectral clustering on hypergraphs.

F. Galasso, Naveen Shankar Nagaraja, T.J. Cardenas, Thomas Brox, B. Schiele
IEEE International Conference on Computer Vision (ICCV), Dec 2013

We further introduced a benchmark for general video segmentation, the VSB-100 benchmark.

IEEE International Conference on Computer Vision (ICCV), 2015

In this paper, we improved the trajectory clustering approach further as we exchanged spectral clustering by a multicut.

IEEE International Conference on Computer Vision (ICCV), 2015

In contrast to the purely unsupervised segmentation approaches above, this approach uses user input to guarantee good segmentations in all cases. We minimize the amount of user effort needed by coupling the user input with automated techniques that exploit the motion of point trajectories and color cues.

Demo videos on motion segmentation from point trajectories

We show some demos on videos from the Berkeley motion segmentation dataset. The left image shows the result after clustering the point trajectories for that video. The right images shows the outcome after color based post-processing, which yields a dense segmentation.


Result from trajectory clustering Dense segmentation Result from trajectory clustering Dense segmentation
High quality video (4MB)

High quality video (23MB)

Result from trajectory clustering Dense segmentation Result from trajectory clustering Dense segmentation
High quality video (8MB)

High quality video (5MB)

Result from trajectory clustering Dense segmentation Result from trajectory clustering Dense segmentation
High quality video (2MB)

High quality video (15MB)

Result from trajectory clustering Dense segmentation Result from trajectory clustering Dense segmentation
High quality video (4MB)

High quality video (9MB)