3D Shape Manipulation Using Deep Generative Adversarial Networks

Abstract

Deep generative models such as Generative Adversarial Networks (GANs) have demonstratedthe ability to effectively learn a manifold over the training data, including 3D data, such thatgenerated objects on this manifold appear crisp and realistic. In the meantime, the process ofcreating detailed, realistic 3D objects by hand is tremendously difficult: it would be incrediblytedious for an unskilled user to create and edit any 3D shape in a realistic fashion. In this work,we propose a voxel-based, comprehensive 3D shape manipulation framework. The frameworkallows users to repeatedly “snap” an imperfect input to a detailed object on the manifold of aGAN, allowing them to create and edit an object with ease. Our framework extends thedefault GAN model by incorporating a projection network and a learned feature space tolearn the snapping operation. We build a shape manipulation application to demonstrate ourresults. Since our main goal is to apply deep learning to a content creation application, wealso assess the general financial impact of 3D deep learning by evaluating whether investorsbehave rationally with respect to deep learning advancements.