He Said, She Said: Deception Detection in Mafia Games Using Word Embeddings and Neural Networks

Abstract

Approaches to automatic and reliable deception detection using verbal cues have shown promising results with recent advances in machine learning and computational linguistics. Deception detection has important applications in border control, Internet fraud and court trials, but research that focus on creating reliable systems suffer from the lack of widely available datasets that mimic realistic situations due to the masked nature of the act. Using de Ruiter and Kachergis’ recent Mafiascum dataset containing over 700 games of a popular party game called Mafia, we performed a statistical analysis on well-known linguistic-based cues to deception in game posts and combined them with pre-trained word embeddings to train baseline machine learning classifiers and neural networks. Our results show significant differences in language used by truthful and deceitful players but also indicate that these cues are context-dependent. Our best classification model is the hybrid CNN-LSTM trained with handcrafted and fastText embeddings, yielding an accuracy of 92.8%, which is approximately 41% higher than human performance. These results contribute to the current body of literature on deception detection and demonstrate the effectiveness of neural networks at the task.