Analyzing, Optimizing and Synthesizing Scenes by Reasoning About Relationships Between Objects

Abstract

3D scene modeling has many applications, including virtual social worlds, massively multiplayer online games, and production of catalog images. However, scene modeling is extremely tedious and challenging, due to the requirement of realism and the involvement of large numbers of objects. Therefore, automated methods for 3D scene modeling are needed. A promising approach is to first analyze the existing 3D scenes, such as the ones in online repositories (e.g. Trimble 3D Warehouse), and then use the knowledge obtained from the analysis step to create new scenes.

Although a significant amount of research has focused on developing algorithms for scene analysis and scene modeling, these processes still remain challenging for two reasons. First, the large variability of 3D scenes makes it hard to capture the commonality among scenes for scene analysis. Second, the highly constrained space of realistic 3D scenes makes it challenging to automatically create satisfactory scenes.

This dissertation pushes the limits of existing efforts on analyzing, synthesizing, and optimizing 3D scenes by reasoning about relationships between objects. First, it describes an algorithm that segments and annotates 3D scenes by considering relationships between objects in a hierarchical representation. Second, it describes a tool that optimizes a 3D scene to produce compositions by considering relationships between objects in the image space and the scene space. Finally, it focuses on style compatibility between objects, which is a relationship that has never been considered in previous scene modeling tools, and it presents a method for learning to predict the stylistic compatibility between 3D furniture models from different object classes.

In this dissertation, we find that relationships between objects are comparatively stable across scenes, and that they can serve as a strong cue for inferring annotation and segmentation of scenes. Furthermore, we also find that modeling relationships between objects helps ensure the realism of synthesized scenes. Therefore, reasoning about relationships between objects greatly facilitates scene analysis and synthesis.