Learning and Deploying Local Features

Abstract

Many computer vision applications including image matching, image-based reconstruction and localization rely on extracting and matching robust local features. A typical local feature pipeline first detects repeatable keypoints in the image (i.e., keypoint detector), and then computes a short vector to uniquely describe each keypoint (i.e., feature descriptor). Both the keypoint detector and the feature descriptor are conventionally hand-crafted based on what is intuitive to the designer. For example, corners or blobs are popular choices of keypoints, and the image gradient is a useful clue for descriptors. However, it is often difficult to define these principles to accommodate various applications. In this thesis, we study data-driven approaches which can more easily tailor the local feature pipeline for target applications. We start with a mobile robotics application that leverages local features extracted from ground texture images to achieve high-precision global localization. The second part of the thesis addresses the problem that existing keypoint detectors that are optimized for natural images suffer from sub-optimal performance on texture images. We therefore learn a keypoint detector specifically for each type of texture using a deep neural network. Our detector automatically learns to identify keypoints that are distinctive in the target texture rather than relying on a set of pre-defined rules. Finally, we focus on a non-parametric approach for learning feature descriptors. Many well-performing local feature descriptors are trained using a triplet loss that includes a tunable margin, which limits its ability to generalize to other types of data and problems. We propose to replace the hard margin with a soft margin that self-tunes as learning progresses. To summarize, we first demonstrate through a novel visual-based localization system where a customized local feature pipeline is critical. Then, we tackle both the keypoint detector and the feature descriptor with generalizable data-driven approaches.