With the capability of modeling bidirectional contexts, denoising autoencoding based
pretraining like BERT achieves better performance than pretraining approaches based on�…

BERT adopts masked language modeling (MLM) for pre-training and is one of the most
successful pre-training models. Since BERT neglects dependency among predicted tokens�…

Natural language understanding has recently seen a surge of progress with the use of
sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which�…

Recently, the pre-trained language model, BERT (and its robustly optimized version
RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and�…

There have been various types of pretraining architectures including autoencoding models
(eg, BERT), autoregressive models (eg, GPT), and encoder-decoder models (eg, T5)�…

Abstract Language model pre-training has proven to be useful in learning universal
language representations. As a state-of-the-art language model pre-training model, BERT�…

Transformer-based pretrained language models (T-PTLMs) have achieved great success in
almost every NLP task. The evolution of these models started with GPT and BERT. These�…

State-of-the-art performance on language understanding tasks is now achieved with
increasingly large networks; the current record holder has billions of parameters. Given a�…

Recent works have shown promising results of prompt tuning in stimulating pre-trained
language models (PLMs) for natural language processing (NLP) tasks. However, to the best�…

Pretraining sentence encoders with language modeling and related unsupervised tasks has
recently been shown to be very effective for language understanding tasks. By�…