SUN RGB-D: An RGB-D Scene Understanding Benchmark Suite

Abstract

Although RGB-D sensors have enabled breakthroughs for several vision tasks, such as 3D reconstruction, we have not attained the same level of success in high-level scene understand- ing. Perhaps one of the main reasons is the lack of a large-scale benchmark with 3D annotations and 3D evaluation metrics. In this paper, we introduce an RGB-D benchmark suite with the goal of advancing the state-of-the-art in all major scene understanding tasks. Our dataset is captured by four different sensors and contains 10,335 RGB-D images, putting it at roughly the same scale as PASCAL VOC. The whole dataset is densely annotated: it includes 146,617 2D polygons and 64,595 3D bounding boxes with accurate object orientations, as well as a 3D room layout and scene category label for each image. This dataset enables us to train data- hungry algorithms for scene understanding tasks, evaluate them using meaningful 3D metrics, avoid overfitting to a small test set, and study cross-sensor bias.