MearcAll papers

arXiv 1902.00751 · cs.LG cs.CL stat.ML

Parameter-Efficient Transfer Learning for NLP

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, and 2 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_01d4e85f
As an alternative, we propose transfer with adapter modules.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_58052438
Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f1465b77
On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_bd89623b
For this, we propose a transfer learning strategy that yields compact and extensible downstream models.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_38ad86ad
We demonstrate on a large and diverse set of text classification tasks that adapters yield parameter-efficient tuning for NLP.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f6dfdb66
We propose a simple yet effective, bottleneck architecture.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_02ecb8df
On the GLUE benchmark, our strategy almost matches the performance of the fully fine-tuned BERT, but uses only 3% task-specific parameters, while fine-tuning uses 100% task-specific parameters.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6f5ea334
In summary, adapter-based tuning yields a single, extensible, model that attains near state-of-the-art performance in text classification.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1c0f6d2b
To achieve these properties, we propose a new bottleneck adapter module.

Adapter tuning for NLP · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_31a9f285
To limit the number of parameters, we propose a bottleneck architecture.

Instantiation for Transformer Networks · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7d7a967e
We show that adapters achieve parameter efficient transfer for text tasks.

Experiments · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f65d66bc
Adapters achieve a mean GLUE score of 80.0, compared to 80.4 achieved by full fine-tuning.

GLUE benchmark · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_0b3c1d55
Unlike the GLUE tasks, there is no comprehensive set of state-of-the-art numbers for this suite of tasks.

We treat MNLIm and MNLImm as separate tasks with individ- · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bb0a30ba
Adapter tuning achieves comparable overall score (80.0) to full fine-tuning (80.4) using 1.3× parameters in total, compared to 9×.

Parameter/Performance trade-off · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a1beedd2
Training adapters with sizes 0.5− 5% of the original model, performance is within 1% of the competitive published results on BERTLARGE.

Parameter/Performance trade-off · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c0f58109
This relates to our observation that for some tasks, fine-tuning only the top layers outperforms full fine-tuning, see Table 2.

Analysis and Discussion · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1902.00751) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.