Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

In a traditional supervised learning system for NLP, we take an input \(\mathbf {x}\) , usually text, and predict an output \(\mathbf {y}\) based on a model \(P(\mathbf {y}|\mathbf {x};\theta)\) . \(\mathbf {y}\) could be a label, text, or other variety of output. To learn the parameters \(\theta\) of this model, we use a dataset containing pairs of inputs and outputs and train a model to predict this conditional probability. We will illustrate this with two stereotypical examples.

First, text classification takes an input text \(\mathbf {x}\) and predicts a label y from a fixed label set \(\mathcal {Y}\) . To give an example, sentiment analysis [94, 128] may take an input \(\mathbf {x}\) = “I love this movie” and predict a label y = ++, out of a label set \(\mathcal {Y}\) = {++, +, ~, -, --}.⁴ Second, conditional text generation takes an input \(\mathbf {x}\) and generates another text \(\mathbf {y}\) . One example is machine translation [59], where the input is text in one language such as the Finnish \(\mathbf {x}\) = “Hyvää huomenta” and the output is the English \(\mathbf {y}\) = “Good morning.”

The main issue with supervised learning is that to train a model \(P(\mathbf {y}|\mathbf {x};\theta)\) , it is necessary to have annotated data for the task, which for many tasks cannot be found in large amounts. Prompt-based learning methods for NLP attempt to circumvent this issue by instead learning an LM that models the probability \(P(\mathbf {x};\theta)\) of text \(\mathbf {x}\) itself (details in Appendix) and using this probability to predict \(\mathbf {y}\) , reducing or obviating the need for large labeled datasets. In this section we lay out a mathematical description of the most fundamental form of prompting, which encompasses many works on prompting and can be expanded to cover others as well. Specifically, basic prompting predicts the highest-scoring \(\hat{\mathbf {y}}\) in three steps.

Now that we have our basic mathematical formulation, we elaborate a few of the basic design considerations that go into a prompting method, which we will elaborate in the following sections:

As noted above, there are two main varieties of prompts: cloze prompts [17, 100], which fill in the blanks of a textual string (e.g., “I love this movie, it is a [Z] movie”), and prefix prompts [67, 71], which continue a string prefix (e.g., “I love this movie. What’s the sentiment of the review? [Z]”). Which one is chosen will depend both on the task and the model that is being used to solve the task. In general, for tasks regarding generation, or tasks being solved using a standard auto-regressive LM, prefix prompts tend to be more conducive, as they mesh well with the left-to-right nature of the model. For tasks that are solved using masked LMs, cloze prompts are a good fit, as they very closely match the form of the pre-training task. Full text reconstruction models are more versatile and can be used with either cloze or prefix prompts. Finally, for some tasks regarding multiple inputs such as text pair classification, prompt templates must contain space for two inputs, [X1] and [X2], or more.

Perhaps the most natural way to create prompts is to manually create intuitive templates based on human introspection. For example, the seminal LAMA dataset [100] provides manually created cloze templates to probe knowledge in LMs. Brown et al. [9] create manually crafted prefix prompts to handle a wide variety of tasks, including question answering, translation, and probing tasks for common sense reasoning. Schick and Schütze [118] and Schick and Schütze [117], 120] use pre-defined templates in a few-shot learning setting on text classification and conditional text generation tasks.

While the strategy of manually crafting templates is intuitive and does allow solving various tasks with some degree of accuracy, there are also several issues with this approach: (1) Creating and experimenting with these prompts is an art that takes time and experience, particularly for some complicated tasks such as semantic parsing [124]; and (2) even experienced prompt designers may fail to manually discover optimal prompts [52].

To address these problems, a number of methods have been proposed to automate the template design process. In particular, the automatically induced prompts can be further separated into discrete prompts, where the prompt is an actual text string, and continuous prompts, where the prompt is instead described directly in the embedding space of the underlying LM.

One other orthogonal design consideration is whether the prompting function \(f_{\text{prompt}}(\mathbf {x})\) is static, using essentially the same prompt template for each input, or dynamic, generating a custom template for each input. Both static and dynamic strategies have been used for different varieties of discrete and continuous prompts, as we will mention below.

The shape of an answer characterizes its granularity. Some common choices include the following:

In practice, how to choose the shape of acceptable answers depends on the task we want to perform. Token or text-span answer spaces are widely used in classification tasks (e.g., sentiment classification; Yin et al. [144]) as well as in other tasks such as relation extraction [100] or named entity recognition [17]. Longer phrasal or sentential answers are often used in language generation tasks [105] and in other tasks such as multiple-choice question answering (where the scores of multiple phrases are compared against each-other; Khashabi et al. [55]).

The next question to answer is how to design the appropriate answer space \(\mathcal {Z}\) , as well as the mapping to the output space \(\mathcal {Y}\) if the answers are not used as the final outputs.

Prompt ensembling is the process of using multiple unanswered prompts for an input at inference time to make predictions. An example is shown in Figure 2(a). The multiple prompts can either be discrete prompts or continuous prompts.⁶ This sort of prompt ensembling can (1) leverage the complementary advantages of different prompts, (2) alleviate the cost of prompt engineering, since choosing one best-performing prompt is challenging, and (3) stabilize performance on downstream tasks.

Prompt ensembling is connected to ensembling methods that are used to combine together multiple systems, which have a long history in machine learning [24, 133, 153]. Current research also borrows ideas from these works to derive effective ways for prompt ensembling, as described below.

Prompt augmentation, also sometimes called demonstration learning [32], provides a few additional answered prompts that can be used to demonstrate how the LM should provide the answer to the actual prompt instantiated with the input \(\mathbf {x}\) . For example, instead of just providing a prompt of “China’s capital is [Z],” the prompt can be prefaced by a few examples such as “Great Britain’s capital is London. Japan’s capital is Tokyo. China’s capital is [Z].” Another example of performing addition of two numbers can be found in Figure 2(b). These few-shot demonstrations take advantage of the ability of strong language models to learn repetitive patterns [9].

Although the idea of prompt augmentation is simple, there are several aspects that make it challenging: (1) Sample Selection: how do we choose the most effective examples? (2) Sample Ordering: How do we properly order the chosen examples?

For those composable tasks, which can be composed based on more fundamental subtasks, we can also perform prompt composition, using multiple sub-prompts, each for one subtask, and then defining a composite prompt based on those sub-prompts. This process is illustrated in Figure 2(c). For example, in the relation extraction task, which aims to extract the relation of two entities, we can break down the task into several subtasks including identifying the characteristics of entities and classifying the relationships between entities. Based on this intuition, Han et al. [41] first use multiple manually created sub-prompts for entity recognition and relation classification and then compose them into a complete prompt based on logic rules for relation extraction.

For tasks where multiple predictions should be performed for one sample (e.g., sequence labeling), directly defining a holistic prompt with regards to the entire input text \(\mathbf {x}\) is challenging. One intuitive method to address this problem is to break down the holistic prompt into different sub-prompts and then answer each sub-prompt separately. Figure 2(d) illustrates this idea with an example from the named entity recognition task, which aims to identify all named entities in an input sentence. In this case, the input will first be converted into a set of text spans, and the model can then be prompted to predict the entity type (including “Not an Entity”) for each span. It is not easy to predict all the span types at the same time due to the large number of spans, so different prompts for each span can be created and predicted separately. This sort of prompt decomposition for named entity recognition has been explored by Cui et al. [17] where they apply the approach we discussed here.

In many cases, prompting methods can be used without any explicit training of the LM for the downstream task, simply taking an LM that has been trained to predict the probability of text \(P(\mathbf {x})\) and applying it as-is to fill the cloze or prefix prompts defined to specify the task. This is traditionally called the zero-shot setting [111], as there is zero training data for the task of interest.

However, there are also methods that use training data to train the model in concert with prompting methods. These consist of either full-data learning, where a reasonably large number of training examples are used to train the model, or few-shot learning [126], where a very small number of examples are used to train the model. Prompting methods are particularly useful in the latter case [9, 32, 117], as there are generally not enough training examples to fully specify the desired behavior, and thus using a prompt to push the model in the right direction is particularly effective.

One thing to note is that for many of the prompt template engineering methods described in Section 3, although annotated training samples are not explicitly used in the training of the downstream task model, they are often used in the construction or validation of the prompts that the downstream task will use. As noted by Perez et al. [96], this is arguably not true zero-shot learning with respect to the downstream task.

In prompt-based downstream task learning, there are usually two types of parameters, namely those from (1) pre-trained LMs and (2) prompts. Which part of parameters should be updated is one important design decision, which can lead to different levels of applicability in different scenarios. We summarize five tuning strategies (as shown in Table 4) based on (i) whether the parameters of the underlying LM are tuned, (ii) whether there are additional prompt-related parameters, and (iii) if there are additional prompt-related parameters, whether those parameters are tuned.

Prompt-based learning has been widely explored in classification-based tasks where prompt templates can be constructed relatively easily, such as text classification [144] and natural language inference [117]. The key to prompting for classification-based tasks is reformulating it as an appropriate prompt. For example, Yin et al. [144] use a prompt such as “the topic of this document is [Z],” which is then fed into mask pre-trained LMs for slot filling.

Unlike classification tasks where cloze questions can often be intuitively constructed, for information extraction tasks constructing prompts often requires more finesse.

There is still a debate⁷ about if deep neural networks are capable of performing “reasoning” or just memorizing patterns based on large training data [3, 89]. As such, there have been a number of attempts to probe models’ reasoning ability by defining benchmark tasks that span different scenarios. We detail below how prompting methods have been used in these tasks.

Question answering (QA) aims to answer a given input question, often based on a context document. QA can take a variety of formats, such as extractive QA (which identifies content from the context document containing the answer; e.g., SQuAD [108]), multiple-choice QA (where the model has to pick from several choices; e.g., RACE [64]), and free-form QA (where the model can return an arbitrary textual string as a response; e.g., NarrativeQA [58]). Generally, these different formats have been handled using different modeling frameworks. One benefit of solving QA problems with LMs, potentially using prompting methods, is that different formats of QA tasks can be solved within the same framework. For example, Khashabi et al. [55] reformulate many QA tasks as a text generation problem by fine-tuning seq2seq-based pre-trained LMs (e.g., T5) and appropriate prompts from the context and questions. Jiang et al. [51] take a closer look at such prompt-based QA systems using sequence-to-sequence pre-trained LMs (T5, BART, and GPT2) and observe that probabilities from these pre-trained LMs on QA tasks are not very predictive of whether the model is correct or not.

Text generation is a family of tasks that involve generating text, usually conditioned on some other piece of information. Prompting methods can be easily applied to these tasks by using prefix prompts together with autoregressive pre-trained LMs. Radford et al. [105] demonstrated impressive ability of such models to perform generation tasks such as text summarization and machine translation using prompts such as “translate to french, [X], [Z].” Brown et al. [9] perform in-context learning (Section 6.2.2) for text generation, creating a prompt with manual templates and augmenting the input with multiple answered prompts. Schick and Schütze [118] explore fixed-prompt LM tuning (Section 6.2.4) for few-shot text summarization with manually crafted templates. [71] investigate fixed-LM prompt tuning (Section 6.2.3) for text summarization and data-to-text generation in few-shot settings, where learnable prefix tokens are prepended to the input while parameters in pre-trained LMs are kept frozen. Dou et al. [23] explored the prompt+LM tuning strategy (Section 6.2.5) on text summarization task, where learnable prefix prompts are used and initialized by different types of guidance signals, which can then be updated together with parameters of pre-trained LMs.

Yuan et al. [147] have demonstrated that prompt learning can be used for automated evaluation of generated texts. Specifically, they conceptualize the evaluation of generated text as a text generation problem, modeled using a pre-trained sequence-to-sequence, and then use prefix prompts that bring the evaluation task closer to the pre-training task. They experimentally find that simply adding the phrase “such as” to the translated text when using pre-trained LMs can lead to a significant improvement in correlation on German–English machine translation evaluation.

There are also a number of applications of prompting techniques that are not NLP tasks in and of themselves but are useful elements of training strong models for any application.

Tsimpoukelli et al. [135] shift the application of prompt learning from text-based NLP to the multi-modal setting (vision and language). Generally, they adopt the fixed-LM prompt tuning strategy together with prompt augmentation techniques. They specifically represent each image as a sequence of continuous embeddings, and a pre-trained LM whose parameters are frozen is prompted with this prefix to generate texts such as image captions. Empirical results show few-shot learning ability: With the help of a few demonstrations (answered prompts), system can rapidly learn words for new objects and novel visual categories.

With plenty of pre-trained LMs to select from (see Appendix), how to choose them to better leverage prompt-based learning is an interesting and difficult problem. So far, there are few to no systematic comparisons of the benefits brought by prompt-based learning for different pre-trained LMs.

As discussed in Section 6, there are a fairly wide variety of methods for tuning parameters of prompts, LMs, or both. However, given the nascent stage of this research field, we still lack a systematic understanding of the tradeoffs between these methods. The field could benefit from systematic explorations such as those performed in the pre-train and fine-tune paradigm regarding the tradeoffs between these different strategies [98].

Despite their success in many scenarios, theoretical analysis and guarantees for prompt-based learning are scarce. Wei et al. [141] showed that soft-prompt tuning can relax the non-degeneracy assumptions (the generation probability of each token is linearly independent) needed for downstream recovery (i.e., recover the ground-truth labels of the downstream task.). making it easier to extract task-specific information. Saunshi et al. [113] verified that text classification tasks can be reformulated as sentence completion tasks, thus making language modeling a meaningful pre-training task. Scao and Rush [114] empirically show that prompting is often worth 100s of data points on average across classification tasks.

Understanding the extent to which prompts are specific to the model and improving the transferability of prompts are also important topics. Reference [96] show that prompts selected under tuned few-shot learning scenario (where one has a larger validation set to choose prompts) generalize well across models of similar sizes while prompts selected under true few-shot learning scenario (where one only has a few training samples) do not generalize as effectively as the former setting among models with similar sizes. The transferability is poor when the model sizes are quite different in both scenarios.

Notably, much of the success of the prompting paradigm is built on top of pre-trained LMs that were developed for the pre-train and fine-tune paradigm, such as BERT. However, are the pre-training methods that are effective for the latter applicable as-is to the former, or can we entirely re-think our pre-training methods to further improve accuracy or ease of applicability to prompting-based learning? This is an important research question that has not been covered extensively by the literature.

Calibration [33] refers to the ability of a model to make good probabilistic predictions. When using the generation probability of the pre-trained LMs (e.g., BART) to predict the answer, we need to be careful, since the probability distribution is typically not well calibrated. Jiang et al. [51] observed the probabilities of pre-trained LMs (e.g., BART, T5, and GPT-2) on QA tasks are well calibrated. Zhao et al. [151] identify three pitfalls (majority label bias, recency bias and common token bias) that lead the pre-trained LMs to be biased toward certain answers when provided answered prompts. For example, if the final answered prompt has a positive label, then this will bias the model toward predicting positive words. To overcome those pitfalls, Zhao et al. [151] first use context-free input (e.g., the prompt would be “Input: Subpar acting. Sentiment: Negative\n Input: Beautiful film. Sentiment: Positive\n Input: N/A. Sentiment:”) to get the initial probability distribution \(P_0\) , then they use the real input (e.g., the prompt would be “Input: Subpar acting. Sentiment: Negative\n Input: Beautiful film. Sentiment: Positive\n Input: Amazing. Sentiment:”) to get the probability distribution \(P_1\) . Finally, these two distributions can be used to get a calibrated generation probability distribution. However, this method has two drawbacks: (1) it comes with the overhead of finding proper context-free input (e.g., whether to use “N/A” or “None”) and (2) the probability distribution of the underlying pre-trained LM is still not calibrated.

Even though we have a calibrated probability distribution, we also need to be careful when we assume a single gold answer for an input. This is because that all surface forms of a same object will compete for finite probability mass [45]. For example, if we consider the gold answer to be “Whirlpool bath,” then the generation probability of it will typically be low, since the word “Bathtub” shares the same meaning and it will take over a large probability mass. To address this issue, we could either (i) perform prompt answer engineering to construct a comprehensive gold answer set using paraphrasing methods (Section 4.2.2) or (ii) calibrate the probability of a word based on its prior likelihood within the context [45].