Mars VR: Reconstructing Interactive 3D Mars Environments for Astronaut Training

Abstract

Human exploration missions to the Moon and Mars are slated for the near future. To prepare, astronauts and mission operations personnel need detailed knowledge of planetary geology to identify areas of interest and to evaluate terrain for rover navigation and extravehicular activity. Currently, Mars rover navigation and terrain surveying is primarily done using satellite data and imagery taken on the Martian surface, however; obtaining data from Mars is expensive, requires a lot of power and bandwidth, and comes with a big opportunity cost. In this project, we investigate ways to extend existing 2D image data to reconstruct 3D Mars environments in which users can perform tasks in an immersive and informative way in Virtual Reality. Using imagery from NASA-JPL's Curiosity rover, we build on existing photogrammetry techniques and use deep learning image colourization to reconstruct high fidelity 3D models that can be used in a real-time graphics engine. Compared to conventional 2D imagery, stereoscopic 3D displays have been demonstrated to be more accurate and efficient for depth-related applications, like judging distances, manipulating objects in space, and remote guidance. With input from the European Space Agency and NASA, we develop a tool set for astronaut training and mission planning applications. Our results demonstrate that our approach could provide the basis for a reconstruction pipeline that is widely applicable to Mars, the Moon and beyond.