MearcAll papers

arXiv 2305.14314 · cs.LG

QLoRA: Efficient Finetuning of Quantized LLMs

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedres_d3e0c53a
We present QL ORA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ee68e978
Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_67a616e8
Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a40392ef
Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2c3d1814
We demonstrate for the first time that it is possible to finetune a quantized 4-bit model without any performance degradation.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2591ee43
QLORA reduces the average memory requirements of finetuning a 65B parameter model from >780GB of GPU memory to <48GB without degrading the runtime or predictive performance compared to a 16bit fully finetuned baseline.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d5ad1b27
Using QLORA, we train the Guanaco family of models, with the second best model reaching 97.8% of the performance level of ChatGPT on the Vicuna [10] benchmark, while being trainable in less than 12 hours on a single consumer GPU; using a single professional GPU over 24 hours we achieve 99.3% with our largest model, essentially closing the gap to ChatGPT on the Vicuna benchmark.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9731b021
When deployed, our smallest Guanaco model (7B parameters) requires just 5 GB of memory and outperforms a 26 GB Alpaca model by more than 20 percentage points on the Vicuna benchmark (Table 6).

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5d481e31
QL ORA introduces multiple innovations designed to reduce memory use without sacrificing performance: (1) 4-bit NormalFloat, an information theoretically optimal quantization data type for normally distributed data that yields better empirical results than 4-bit Integers and 4-bit Floats.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cca1fce1
In addition to showing that QL ORA recovers 16-bit performance (§4) and training a state-of-the-art chatbot, Guanaco, (§5), we also analyze trends in the trained models.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c75a3568
First, we find that data quality is far more important than dataset size, e.g., a 9k sample dataset (OASST1) outperformed a 450k sample dataset (FLAN v2, subsampled) on chatbot performance, even when both are meant to support instruction following generalization.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f96fa7c9
Second, we show that strong Massive Multitask Language Understanding (MMLU) benchmark performance does not imply strong Vicuna chatbot benchmark performance and vice versa—in other words, dataset suitability matters more than size for a given task.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_dbdd684f
We find that GPT-4 and human evaluations largely agree on the rank of model performance in the tournaments, but we also find there are instances of strong disagreement.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7d0412b7
QL ORA improves over LoRA by quantizing the transformer model to 4-bit precision and using paged optimizers to handle memory spikes.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_df2fa534
QL ORA achieves high-fidelity 4-bit finetuning via two techniques we propose—4-bit NormalFloat (NF4) quantization and Double Quantization.

QL ORA Finetuning · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_07092a76
4 QLoRA vs. Standard Finetuning We have discussed how QLoRA works and how it can significantly reduce the required memory for finetuning models.

QL ORA Finetuning · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_49f1da04
We improve on the Stanford Alpaca fully finetuned default hyperparameters to construct a strong 16-bit baseline for comparisons.

Model · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_0a6b17e8
As shown in Figure 2 for LLaMA 7B finetuning on Alpaca, we find that the most critical LoRA hyperparameter is how many LoRA adapters are used in total and that LoRA on all linear transformer block layers are required to match full finetuning performance.

Model · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_01e8ec2d
The NormalFloat data type significantly improves the bit-for-bit accuracy gains compared to regular 4-bit Floats.

Model · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_17642bb7
Similarly, we find that default hyperparameters for fully finetuned baselines are undertuned.

Model · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_118d27f0
We do small hyperparameter searches for the 13B and 33B LLaMA models and we find that all hyperparameter settings found at 7B generalize (including number of epochs) except learning rate and batch size.

Experimental setup · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fcf57f80
Note this relative score can be higher than 100% if the model achieves a higher absolute score than ChatGPT.

Evaluation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_191707f4
We find a significant ordering effect with GPT-4 increasing the score of the response occurring earlier in the prompt.

Evaluation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_acd3f9d2
Human Evaluation While recent work indicates generative models can be effectively employed for system evaluations [19], the reliability GPT-4 ratings to assess chatbot performance is, to our knowledge, yet to be proven to correlate with human judgments.

Evaluation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_131c2c98
Based on our automated and human evaluations, we find that the top QL ORA tuned model, Guanaco 65B, which we finetune on a variant of OASST1, is the best-performing open-source chatbot model and offers performance competitive to ChatGPT.

Guanaco: QL ORA trained on OASST1 is a State-of-the-art Chatbot · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9fb05803
We find that Guanaco 65B is the best-performing model after GPT-4, achieving 99.3% performance relative to ChatGPT.

Guanaco: QL ORA trained on OASST1 is a State-of-the-art Chatbot · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cbc2f1e9
Guanaco 33B has more parameters than the Vicuna 13B model, but uses only 4-bit precision for its weights and is thus much more memory efficient at 21 GB vs 26 GB, providing a three percentage points of improvement over Vicuna 13B.

Guanaco: QL ORA trained on OASST1 is a State-of-the-art Chatbot · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8d400604
Elo rankings in Table 7 indicate that Guanaco 33B and 65B models outperform all models besides GPT-4 on the Vicuna and OA benchmarks and that they perform comparably to ChatGPT in line with Table 6.

Guanaco: QL ORA trained on OASST1 is a State-of-the-art Chatbot · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1c08fcbe
So the unique factorization of 1833 is: 2ˆ1 * 3ˆ2 * 17ˆ1 I hope this helps!

Qualitative Analysis of Example Generations · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_323a819b
When manually comparing generations from ChatGPT and Guanaco 65B on the Vicuna benchmark, we find that subjective preferences start to play an important role as the authors of this paper disagreed on the many preferred responses.

Considerations · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_310b92b5
In addition, in Table 7, we observe that GPT-4 assigns significantly higher scores to its own outputs compared to human ratings, Elo of 1348 vs 1176, which represent an additional 20% probability of winning against an opponent.

Considerations · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b228e6fb
The authors show that models trained on this collection outperform other public collections including the original FLAN 2021 [ 62], T0++ [ 50], Super-Natural Instructions [ 60], and OPT-IML [ 29].

Datasets · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7c1b6353
We find hyperparameters to be largely robust across datasets.

Hyperparameters · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1e7432f0
We find that increasing the dataset size and increasing the number of epochs improves MMLU only marginally (0.0 - 0.5 MMLU), while the difference between datasets is up to 40x larger (1.5 - 8.0 MMLU).

Ablations · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1ead1e34
While increasing the dataset size and training for more than 1 epochs helps with MMLU performance, the difference between datasets are far larger, indicating that dataset quality affects MMLU performance more than dataset size.

Normality of Trained Neural Network Weights · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7a2e348b
We find that the weights of each hidden unit have different normal distributions.

Normality of Trained Neural Network Weights · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2305.14314) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.