Learning Sentence Representations with Recurrent Neural Networks

Abstract

Recognizing the limits of the existing semantic vector compositional models, we propose a novel recurrent neural network (RNN) language model that learns vector representations for sentences. Our method uses a small-scale sigmoid RNN model that takes word vector sequences as input and outputs a serial recall of the word vectors and the sigmoid hidden unit representations for the next sequences. We provide empirical evidence to suggest that the RNN sequential recall and next sequence prediction model is sensitive to the similarity of the input vector sequences and produces meaningful sigmoidsensitive to the similarity of the input vector sequences and produces meaningful sigmoid hidden unit representations that contain information regarding the words and the sentiment of each sequence. The results suggest several interesting possibilities for future work in extending the model to a full-scale recurrent neural language model that learns compositional vector representations for sentences. Keywords: language model, recurrent neural networks, compositional semantics