A Non-Autoregressive Language Variational Autoencoder

Standard recurrent neural network language models generate sentences one word at a time and do not work from an explicit global sentence representation. Language models based on the Variational Autoencoder (VAE) generative model incorporate distributed latent representation of the entire sequence, and then, like a standard neural network, autoregressively produces output. This property allows it to explicitly model the holistic properties of sentences such as style, topic, and syntax. Non-autoregressive models, on the other hand, generate all tokens in one pass, which leads to increased efficiency through parallel processing and text generation. However, directly modeling the joint distribution of all tokens simultaneously is challenging, and even with increasingly complex model structures. This paper introduced generative models with autoregressive properties and non-autoregressive properties, which generate all tokens in one pass. We showed how non-autoregressive models differ from autoregressive ones in structure, proposed several model architectures, and compared their performance on several text datasets. In addition, we examined problems of Variational Autoencoder generative models in training.