Abstract Deep Neural Networks (DNNs) are powerful models that have achieved excellent
performance on difficult learning tasks. Although DNNs work well whenever large labeled�…

The prevalent approach to sequence to sequence learning maps an input sequence to a
variable length output sequence via recurrent neural networks. We introduce an architecture�…

We present two approaches to use unlabeled data to improve Sequence Learningwith
recurrent networks. The first approach is to predict what comes next in asequence, which is�…

Sequence-to-sequence learning with neural networks has become the de facto standard for
sequence modeling. This approach typically models the local distribution over the next�…

This paper introduces Grid Long Short-Term Memory, a network of LSTM cells arranged in a
multidimensional grid that can be applied to vectors, sequences or higher dimensional data�…

The Softmax function is used in the final layer of nearly all existing sequence-to-sequence
models for language generation. However, it is usually the slowest layer to compute which�…

There has been much recent work on training neural attention models at the sequence-level
using either reinforcement learning-style methods or by optimizing the beam. In this paper�…

We present the Compressive Transformer, an attentive sequence model which compresses
past memories for long-range sequence learning. We find the Compressive Transformer�…

In sequence to sequence learning, the self-attention mechanism proves to be highly
effective, and achieves significant improvements in many tasks. However, the self-attention�…

In this paper, we explore the possibility of leveraging Residual Networks (ResNet), a
powerful structure in constructing extremely deep neural network for image understanding�…