clm_fbb96175We propose Low-Rank Adaptation, or LoRA, which freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of … (excerpt)
arXiv 2106.09685 · cs.CL cs.AI cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_fbb96175We propose Low-Rank Adaptation, or LoRA, which freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of … (excerpt)
res_6557dffc1While GPT-3 175B achieves non-trivial performance with few-shot learning, fine-tuning boosts its performance significantly as shown in Appendix A.
clm_a1fbe141Using GPT-3 175B as an example, we show that a very low rank (i.e.,r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) … (excerpt)
clm_32b4f107The task of finding ∆Φ thus becomes optimizing over Θ: max Θ ∑ (x,y)∈Z |y|∑ t=1 log ( pΦ0+∆Φ(Θ)(yt|x,y<t) ) (2) In the subsequent sections, we propose to use a … (excerpt)
clm_b8119e86LoRA outperforms several baselines with comparable or fewer trainable parameters.
res_02e34af7We focus our study on GPT-3 175B, where we achieved the largest reduction of trainable parameters (up to 10,000×) without adversely affecting task performances.
clm_b7cedaa1We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient.
clm_cd15a214We propose LoRA, an efficient adaptation strategy that neither introduces inference latency nor reduces input sequence length while retaining high model quality.
res_cdc8f02b1) LoRA can be combined with other efficient adaptation methods, potentially providing orthogonal improvement.
res_e77e544bAs shown in Table 8, fine-tuning improves the model performance drastically compared to few-shot learning on datasets large and small.
res_e35e6553Larger batch size and sequence length help to mitigate the latency, but the slow-down can be as high as over 30% in an online, short-sequence-length scenario.
clm_4776b757First of all, LoRA+PE significantly outperforms both LoRA and prefix-embedding tuning on WikiSQL, which indicates that LoRA is somewhat orthogonal to prefix-embedding tuning.
res_59bb6c5aTo our surprise, PrefixEmbed and PrefixLayer performs very poorly on MNLI-100 dataset, with PrefixEmbed performing only slightly better than random chance (37.6% vs. 33.3%).
res_9951bb64LoRA achieves better performance than fine-tuning on both MNLI-100 and MNLI-Full, and comparable results on MNLI-1k and MNLI-10K considering the (±0.3) variance due to random seeds.
res_a00eaca9Using the E2E NLG Challenge dataset as an example, we report the validation loss and test metrics achieved by different choices ofr after training for 26,000 steps.
res_5c58d408Note again that ∆W does not contain the top singular directions ofW , since the similarity between the top 4 directions in ∆W and the top-10% of those inW barely … (excerpt)
res_20bf30c3In other words, there are (generally speaking) four feature directions in each layer (out of the entire feature space from the pre-trained modelW ), that need to be amplified by … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2106.09685) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.