res_dc724e41Our end-to-end learned approach outperforms GPT-3’s few-shot learning by a large margin.
arXiv 2104.08691 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
res_dc724e41Our end-to-end learned approach outperforms GPT-3’s few-shot learning by a large margin.
clm_8af0ff8fMore remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method “closes the gap” … (excerpt)
clm_cabcec28Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient “prompt ensembling.”
res_da7a4728Our approach significantly outperforms fewshot prompt design using GPT-3.
clm_ef2595daWhile this technique outperforms manual prompt design, there is still a gap relative to model tuning.
clm_7c24fa8eIn this paper, we propose prompt tuning as a further simplification for adapting language models.
res_a2e884f3This “soft prompt” is trained end-to-end and can condense the signal from a full labeled dataset, allowing our method to outperform few-shot prompts and close the quality gap with model … (excerpt)
clm_5d3baddeThrough detailed experiments in sections 2–3, we demonstrate that language model capacity is a key ingredient for these approaches to succeed.
res_77ecdb94We show in Section 5 that by capturing the task definition in the prompt while keeping the generalist parameters fixed, we are able to achieve better resilience to domain shifts.
clm_a376aeb9In Section 6, we show that “prompt ensembling”, learning multiple prompts for the same task, can boost quality and is more efficient than classic model ensembling.
res_475ae284We leverage the public T5.1.1 checkpoints, which include improvements over the original T5.1 Our “default” configuration, plotted with a green ‘×’ ( ) throughout, uses an LM-adapted version of T5 … (excerpt)
res_dc30a1ebWhile this is longer than the default 10-token prefix used by Li and Liang (2021), our method still uses fewer task-specific parameters, as we only tune the input layer, as … (excerpt)
res_6c0ea66cWe train our prompts for30,000 steps using T5’s standard cross-entropy loss, with a constant learn- 1These improvements are (1) the removal of all supervised data from pre-training, (2) adjustments to … (excerpt)
res_3f20a55c(1) “Model Tuning”: For an apples-to-apples comparison, we tune on each task separately, as in our prompt tuning setup.3 (2) “Model Tuning (Multitask)”: We use T5’s multi-task tuning setup to … (excerpt)
res_baf8eebe1), we see that prompt tuning becomes more competitive with model tuning as scale increases.
res_937ac01a3To improve this baseline, we performed a sweep over the batch size hyperparameter and selected 216 tokens per batch.
clm_c87875e3(c)Pre-training objective: LM adaptation outperforms span corruption, even when a sentinel is added to downstream task targets, but XXL works well with any method.
res_9a074452Figure 3(a) shows that for most model sizes, increasing prompt length beyond a single token is critical to achieve good performance.
clm_f3bd6b42In the non-optimal “span corruption” setting, we observe instability across model sizes, with the Small model outperforming the larger Base, Large, and XL models.
res_11aef410Furthermore, this poor performance is not due to random variance in prompt tuning, as we observe low variance across 3 runs for each size.
res_ed2d47cd10As another difference, P-tuning requires the addition of “anchor” tokens in the input (e.g. a question mark following the hypothesis in the RTE task) to achieve strong performance, while prompt … (excerpt)
res_8818cdeaFor our experiments, we train on SQuAD (Rajpurkar et al., 2016) and evaluate on each of the out-of-domain datasets.11 Table 1 shows that prompt tuning outperforms model tuning on the … (excerpt)
res_d0b35aecTable 2 shows that training a lightweight prompt on the QQP data and evaluating on MRPC gives much better performance than tuning the entire 11We select checkpoints based on SQuAD … (excerpt)
clm_2b903898On zeroshot domain transfer, we found that prompt tuning leads to improved generalization.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2104.08691) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.