MearcAll papers

arXiv 2101.00190 · cs.CL

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Xiang Lisa Li, Percy Liang

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_cb5ccf0a
In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen, but optimizes a small continuous task-specific vector (called … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d97dc262
We find that by learning only 0.1% of the parameters, prefix-tuning obtains comparable performance in the full data setting, outperforms fine-tuning in low-data settings, and extrapolates better to examples with … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f6297584
We propose prefix-tuning (bottom), which freezes the Transformer parameters and only optimizes the prefix (the red prefix blocks).

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_890d8293
In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation (NLG) tasks, inspired by prompting.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dab2f56b
In lowdata settings, prefix-tuning on average outperforms fine-tuning on both tasks (§6.3).

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_78266537
(2) 4 Prefix-Tuning We propose prefix-tuning as an alternative to fine-tuning for conditional generation tasks.

Method: Fine-tuning · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a193241e
We find that adding only 0.1% task-specific parameters,9 prefix-tuning is effective in table-to-text generation, outperforming other lightweight baselines (ADAPTER and FT-TOP 2) and achieving a comparable performance with fine-tuning.

Table-to-text Generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a77463a3
Table 1 shows that prefix-tuning is significantly better than ADAPTER (0.1%), attaining 4.1 BLEU improvement per dataset on average.

Table-to-text Generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e209c46f
Even when we compare with fine-tuning (100%) and adapter-tuning (3.0%), which update significantly more parameters than prefix-tuning, prefixtuning still achieves results comparable or better than those two systems.

Table-to-text Generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e7857209
As shown in Table 2, with 2% parameters, prefixtuning obtains slightly lower performance than fine- 9250K for E2E, 250K for WebNLG, and 500K for DART vs. 345M GPT-2 parameters.

Summarization · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bdd36d99
Thus, we average over 10 models to get an estimate for each low-data setting.11 Figure 3 (right) shows that prefix-tuning outperforms fine-tuning in low-data regimes by2.9 BLEU on average, in … (excerpt)

Low-data Setting · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f8fa641f
With only0.1% parameters, Prefix-tuning outperforms other lightweight baselines and achieves a comparable performance with fine-tuning.

Extrapolation · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_aba755fa
(Right) prefix-tuning (orange) outperforms fine-tuning (blue) in low-data regimes in addition to requiring many fewer parameters.

method · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1621bd58
Prefixtuning outperforms fine-tuning on both news-to-sports and within-news splits. and test on sports articles.

method · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_58e08f1e
We also find that adapter-tuning achieves good extrapolation performance, comparable with prefixtuning, as shown in Table 1.

method · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0ffa6456
Initializing the prefix with activations of real words significantly improves generation, as shown in Figure 5.

Initialization · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8139722b
We discover that despite learning 1000x fewer parameters than finetuning, prefix-tuning can maintain a comparable performance in a full data setting and outperforms fine-tuning in both low-data and extrapolation settings.

Conclusion · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_68961e79
It validates that random initialization (from a uniform (0,1) distirbution) significantly underperforms initializing with real words; Additionally, initializing with task-relevant words (e.g., “summarization” and “table-to-text”) attains slightly better generation scores … (excerpt)

Additional Results for the Initialization · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2101.00190) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.