Robust Semantic Segmentation using Deep Fusion

Abstract: Robust semantic scene understanding of unstructured environments is critical for robots operating in the real world. Several inherent natural factors such as shadows, glare and snow make this problem highly challenging, especially using RGB images. In this paper, we propose the use of multispectral and multimodal images to increase robustness of segmentation in real-world outdoor environments. Deep Convolutional Neural Network (DCNN) architectures define the state of the art in various segmentation tasks, however, architectures that incorporate fusion has not been sufficiently explored. We explore early and late fusion architectures for dense pixel-wise segmentation from RGB, Near-InfraRed (NIR) channels, and depth data. We identify data augmentation strategies that enable training of very deep fusion models using small datasets. We qualitatively and quantitatively evaluate our approach and show it exceeds several other state of the art architectures. In addition, we present experimental results for segmentation under challenging real-world conditions. Demo and dataset is publicly available at http://deepscene.cs.uni-freiburg.de.