Fast and Robust 3D Reconstruction

Abstract

3D reconstruction from visual data is an important subtask for robotics, autonomous machines, and 3D scene understanding. It involves estimating camera and object motion from images/video, as well as 3D structure. I will introduce an approach known as Optimization Guided Neural Iterations (OGNI), and demonstrate how it can be applied to various 3D reconstruction tasks. In OGNI-based approaches, we mimic classical optimization algorithms by breaking down each task into a series of small revisions predicted by a shallow network. Each revision is supervised independently, and is informed by features conditioned on a running estimate of the solution. This mechanism is suprisingly general, and leads to robust and efficient solutions to 3D reconstruction. Moreover, I introduce several explicit optimization layers which enable one to reformat these challenging problems into easier, low-level vision tasks. On visual SLAM, stereo matching and object pose estimation, I show how this approach leads to state-of-the art accuracy and/or speed. I also discuss potential future directions for this line of work.