arXiv 2101.00190 · cs.CL
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Xiang Lisa Li, Percy Liang
Read on arXiv · arXiv non-exclusive license
- No reproduction recorded 18
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen, but optimizes a small continuous task-specific vector (called … (excerpt)
Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that by learning only 0.1% of the parameters, prefix-tuning obtains comparable performance in the full data setting, outperforms fine-tuning in low-data settings, and extrapolates better to examples with … (excerpt)
Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
We propose prefix-tuning (bottom), which freezes the Transformer parameters and only optimizes the prefix (the red prefix blocks).
Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation (NLG) tasks, inspired by prompting.
Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
In lowdata settings, prefix-tuning on average outperforms fine-tuning on both tasks (§6.3).
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
(2) 4 Prefix-Tuning We propose prefix-tuning as an alternative to fine-tuning for conditional generation tasks.
Method: Fine-tuning · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that adding only 0.1% task-specific parameters,9 prefix-tuning is effective in table-to-text generation, outperforming other lightweight baselines (ADAPTER and FT-TOP 2) and achieving a comparable performance with fine-tuning.
Table-to-text Generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions
Table 1 shows that prefix-tuning is significantly better than ADAPTER (0.1%), attaining 4.1 BLEU improvement per dataset on average.
Table-to-text Generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions
Even when we compare with fine-tuning (100%) and adapter-tuning (3.0%), which update significantly more parameters than prefix-tuning, prefixtuning still achieves results comparable or better than those two systems.
Table-to-text Generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions
As shown in Table 2, with 2% parameters, prefixtuning obtains slightly lower performance than fine- 9250K for E2E, 250K for WebNLG, and 500K for DART vs. 345M GPT-2 parameters.
Summarization · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions
Thus, we average over 10 models to get an estimate for each low-data setting.11 Figure 3 (right) shows that prefix-tuning outperforms fine-tuning in low-data regimes by2.9 BLEU on average, in … (excerpt)
Low-data Setting · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions
With only0.1% parameters, Prefix-tuning outperforms other lightweight baselines and achieves a comparable performance with fine-tuning.
Extrapolation · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
(Right) prefix-tuning (orange) outperforms fine-tuning (blue) in low-data regimes in addition to requiring many fewer parameters.
method · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
Prefixtuning outperforms fine-tuning on both news-to-sports and within-news splits. and test on sports articles.
method · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
We also find that adapter-tuning achieves good extrapolation performance, comparable with prefixtuning, as shown in Table 1.
method · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
Initializing the prefix with activations of real words significantly improves generation, as shown in Figure 5.
Initialization · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions
We discover that despite learning 1000x fewer parameters than finetuning, prefix-tuning can maintain a comparable performance in a full data setting and outperforms fine-tuning in both low-data and extrapolation settings.
Conclusion · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions
It validates that random initialization (from a uniform (0,1) distirbution) significantly underperforms initializing with real words; Additionally, initializing with task-relevant words (e.g., “summarization” and “table-to-text”) attains slightly better generation scores … (excerpt)
Additional Results for the Initialization · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions