Zero-shot Semantic Segmentation

Abstract

Visual recognition systems struggle to identify uncommon classes not captured in real-world image datasets because annotation labels are limited to finite sets of common objects. This motivates the need to develop vision systems that can infer the identity of previously unseen objects from an open vocabulary. To that end, we propose a new task, zero-shot semantic segmentation, in which each pixel is assigned a label from a either a set of seen classes (observed in the training dataset) or unseen classes (not observed in the training dataset). To attempt this task, we propose the Seenmask Zero-shot Network (SZN), a deep learning model that aligns visual and semantic embedding encodings for pixel-label pairs in a joint-embedding space. By projecting pixels from unseen object classes into this joint-embedding space and selecting nearby semantic embeddings, we can infer new labels. We also propose a seenmask mechanism that allows the model to determine whether a given pixel’s label belongs among the seen or unseen labels for a more intelligent inference approach. Our results demonstrates the importance of having high class density for properly aligning visual and semantic embeddings in a joint-embedding space. Our analysis also reveals the theoretical upper bound on accuracy if the seenmask mechanism were perfect, revealing the need for more work on this approach.