Learning Word and Sentence Embeddings Using a Generative Convolutional Network

Abstract

In recent years, sentence modeling using dense vector representations has been a central concern in Natural Language Processing research. While many efforts are essentially focused on the quality of the embeddings in downstream classification tasks, our contribution focuses on the understanding of new forms of computing word representations using generative architectures based on 2D Convolutional Neural Networks. We treat a sentence as a $n \times m$ input image, such that it can be processed using 2D convolutional operations. In contrast to similar current approaches, where the input image remains untouched along the whole learning process, our contribution proposes the use of the learned 2D convolutional filters for modifying the input arrays in order to compute the corresponding word and sentence vector representations at once. We also propose to compute word dictionaries for local contexts and a global dictionary to fuse every word local meaning in a single representation. We call this proposed model a Word Embedding Generative Convolutional Network (WEGCN). Our experiments show that our method is capable of jointly estimating consistent word and sentence embeddings, thus opening pathways for future research in this vein.