MearcAll papers

arXiv 2104.08691 · cs.CL

The Power of Scale for Parameter-Efficient Prompt Tuning

Brian Lester, Rami Al-Rfou, Noah Constant

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedres_dc724e41
Our end-to-end learned approach outperforms GPT-3’s few-shot learning by a large margin.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8af0ff8f
More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method “closes the gap” … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cabcec28
Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient “prompt ensembling.”

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_da7a4728
Our approach significantly outperforms fewshot prompt design using GPT-3.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ef2595da
While this technique outperforms manual prompt design, there is still a gap relative to model tuning.

Model · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7c24fa8e
In this paper, we propose prompt tuning as a further simplification for adapting language models.

Model · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a2e884f3
This “soft prompt” is trained end-to-end and can condense the signal from a full labeled dataset, allowing our method to outperform few-shot prompts and close the quality gap with model … (excerpt)

Model · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5d3badde
Through detailed experiments in sections 2–3, we demonstrate that language model capacity is a key ingredient for these approaches to succeed.

Model · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_77ecdb94
We show in Section 5 that by capturing the task definition in the prompt while keeping the generalist parameters fixed, we are able to achieve better resilience to domain shifts.

Model · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a376aeb9
In Section 6, we show that “prompt ensembling”, learning multiple prompts for the same task, can boost quality and is more efficient than classic model ensembling.

Model · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_475ae284
We leverage the public T5.1.1 checkpoints, which include improvements over the original T5.1 Our “default” configuration, plotted with a green ‘×’ ( ) throughout, uses an LM-adapted version of T5 … (excerpt)

Results · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dc30a1eb
While this is longer than the default 10-token prefix used by Li and Liang (2021), our method still uses fewer task-specific parameters, as we only tune the input layer, as … (excerpt)

Results · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6c0ea66c
We train our prompts for30,000 steps using T5’s standard cross-entropy loss, with a constant learn- 1These improvements are (1) the removal of all supervised data from pre-training, (2) adjustments to … (excerpt)

Results · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3f20a55c
(1) “Model Tuning”: For an apples-to-apples comparison, we tune on each task separately, as in our prompt tuning setup.3 (2) “Model Tuning (Multitask)”: We use T5’s multi-task tuning setup to … (excerpt)

Closing the Gap · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_baf8eebe
1), we see that prompt tuning becomes more competitive with model tuning as scale increases.

Closing the Gap · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_937ac01a
3To improve this baseline, we performed a sweep over the batch size hyperparameter and selected 216 tokens per batch.

Closing the Gap · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c87875e3
(c)Pre-training objective: LM adaptation outperforms span corruption, even when a sentinel is added to downstream task targets, but XXL works well with any method.

Closing the Gap · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9a074452
Figure 3(a) shows that for most model sizes, increasing prompt length beyond a single token is critical to achieve good performance.

Ablation Study · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f3bd6b42
In the non-optimal “span corruption” setting, we observe instability across model sizes, with the Small model outperforming the larger Base, Large, and XL models.

Ablation Study · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_11aef410
Furthermore, this poor performance is not due to random variance in prompt tuning, as we observe low variance across 3 runs for each size.

Ablation Study · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ed2d47cd
10As another difference, P-tuning requires the addition of “anchor” tokens in the input (e.g. a question mark following the hypothesis in the RTE task) to achieve strong performance, while prompt … (excerpt)

Comparison to Similar Approaches · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8818cdea
For our experiments, we train on SQuAD (Rajpurkar et al., 2016) and evaluate on each of the out-of-domain datasets.11 Table 1 shows that prompt tuning outperforms model tuning on the … (excerpt)

Resilience to Domain Shift · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d0b35aec
Table 2 shows that training a lightweight prompt on the QQP data and evaluating on MRPC gives much better performance than tuning the entire 11We select checkpoints based on SQuAD … (excerpt)

Resilience to Domain Shift · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2b903898
On zeroshot domain transfer, we found that prompt tuning leads to improved generalization.

Conclusion · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2104.08691) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.