MearcAll papers

arXiv 2106.09685 · cs.CL cs.AI cs.LG

LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and 2 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_fbb96175
We propose Low-Rank Adaptation, or LoRA, which freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of … (excerpt)

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6557dffc
1While GPT-3 175B achieves non-trivial performance with few-shot learning, fine-tuning boosts its performance significantly as shown in Appendix A.

I NTRODUCTION · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a1fbe141
Using GPT-3 175B as an example, we show that a very low rank (i.e.,r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) … (excerpt)

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_32b4f107
The task of finding ∆Φ thus becomes optimizing over Θ: max Θ ∑ (x,y)∈Z |y|∑ t=1 log ( pΦ0+∆Φ(Θ)(yt|x,y<t) ) (2) In the subsequent sections, we propose to use a … (excerpt)

P ROBLEM STATEMENT · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b8119e86
LoRA outperforms several baselines with comparable or fewer trainable parameters.

B ASELINES · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_02e34af7
We focus our study on GPT-3 175B, where we achieved the largest reduction of trainable parameters (up to 10,000×) without adversely affecting task performances.

U NDERSTANDING THE LOW-RANK UPDATES · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b7cedaa1
We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient.

U NDERSTANDING THE LOW-RANK UPDATES · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cd15a214
We propose LoRA, an efficient adaptation strategy that neither introduces inference latency nor reduces input sequence length while retaining high model quality.

C ONCLUSION AND FUTURE WORK · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cdc8f02b
1) LoRA can be combined with other efficient adaptation methods, potentially providing orthogonal improvement.

C ONCLUSION AND FUTURE WORK · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e77e544b
As shown in Table 8, fine-tuning improves the model performance drastically compared to few-shot learning on datasets large and small.

L ARGE LANGUAGE MODELS STILL NEED PARAMETER UPDATES · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e35e6553
Larger batch size and sequence length help to mitigate the latency, but the slow-down can be as high as over 30% in an online, short-sequence-length scenario.

I NFERENCE LATENCY INTRODUCED BY ADAPTER LAYERS · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4776b757
First of all, LoRA+PE significantly outperforms both LoRA and prefix-embedding tuning on WikiSQL, which indicates that LoRA is somewhat orthogonal to prefix-embedding tuning.

C OMBINING LORA WITH PREFIX TUNING · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_59bb6c5a
To our surprise, PrefixEmbed and PrefixLayer performs very poorly on MNLI-100 dataset, with PrefixEmbed performing only slightly better than random chance (37.6% vs. 33.3%).

L OW-DATA REGIME · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9951bb64
LoRA achieves better performance than fine-tuning on both MNLI-100 and MNLI-Full, and comparable results on MNLI-1k and MNLI-10K considering the (±0.3) variance due to random seeds.

L OW-DATA REGIME · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a00eaca9
Using the E2E NLG Challenge dataset as an example, we report the validation loss and test metrics achieved by different choices ofr after training for 26,000 steps.

E FFECT OF r ON GPT-2 · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5c58d408
Note again that ∆W does not contain the top singular directions ofW , since the similarity between the top 4 directions in ∆W and the top-10% of those inW barely … (excerpt)

E FFECT OF r ON GPT-2 · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_20bf30c3
In other words, there are (generally speaking) four feature directions in each layer (out of the entire feature space from the pre-trained modelW ), that need to be amplified by … (excerpt)

A MPLIFICATION FACTOR · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2106.09685) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.