ORIGINAL

Abstract

Human activity recognition has been a breakthrough with implications in smart environments, healthcare, and global security. Previous work has relied on either remote sensing such as images and videos or Physically Integrated sensing including location and biometrics to design predictive models for human activity. Even when multiple sensor modalities are employed in the same space, modern human activity recognition methods lack meaningful ways to integrate sensor modalities while preserving each respective sensor's structured data for enhanced performance. In this study we consider a sub-class of human activity detection problems based on human-object interactions and use a simulated home environment dataset with vision data in the form of 400-by-400 pixel images and physically integrated (PI) sensing data from 2-dimensional real-world coordinates of humans and key objects. By combining both modalities, we propose a sensor fusion approach for deep learning. We diverge from previous work in attempting to preserve spatial qualities from both modalities in the model training and creation. We present numerous approaches for Physically Integrated sensor information representation as spatial maps and train a fusion-based deep convolutional neural network model that predicts human activity with more than 90% accuracy with 6.96 times fewer data required than a model trained only on image data. We also show that the optimal insertion point of PI spatial maps is prior to the last convolutional layer of a neural network only and this model trains similarly well as inserting PI maps at all hidden convolutional layers.