Virtual Environments as Driving Schools for Deep Learning Vision-Based Sensors in Self-Driving Cars

Abstract

At the turn of the 20th century, inventors and industrialists alikestrived to enable every person to own and drive a car. Overtime, automobileownership grew to meet that vision. One hundred years later, automobilemanufacturers and technology companies are working on self-drivingcars which would be neither owned nor driven by individuals. The benefitsof replacing cars with fully autonomous vehicles are enormous. Whileit is difficult to put a value on lives saved, injuries avoided, pollutionreduced, and commute time repurposed, economic savings from this technologyare estimated to be on the order of trillions of dollars. The mainroadblock in achieving the vision for this century is developing technologywhich would enable autonomous vehicles to perceive and understandthe environment as well as, if not better than, human divers. Perceptionis a roadblock because presently no algorithm is capable of reachinghuman levels of cognition.This thesis explores the interaction between virtual reality simulationand Deep Learning which may develop computer vision that rivals humanvision. The specific problem considered is detection and localizationof a stop object, the stop sign, based on an image. A video game,Grand Theft Auto 5, is used to collect over half a million imagesand corresponding ground truth labels with and without stop signsin various lighting and weather conditions. A deep convolutional neuralnetwork trained on this data and fine tuned on real world data achievesaccuracy in stop sign detection of over 95\% within 20 meters of thestop sign and has a false positive rate of 4\% on test data from thereal world. Additionally, the physical constraints on this problemare analysed, a framework for the use of simulators is developed,and domain adaptation and multi-task learning are explored.