MearcAll papers

arXiv 2302.13971 · cs.CL

LLaMA: Open and Efficient Foundation Language Models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, and 8 more

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedclm_57ec884b
We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1b3b56c4
In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA- 65B is competitive with the best models, Chinchilla-70B and PaLM-540B.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7f6eec9d
For instance, although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9ceb35b4
For instance, LLaMA-13B outperforms GPT-3 on most benchmarks, despite being 10× smaller.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ce8e39a7
2.4 Efficient implementation We make several optimizations to improve the training speed of our models.

Optimizer · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2c9f3d82
First, LLaMA-65B outperforms Chinchilla-70B on all reported benchmarks but BoolQ.

Common Sense Reasoning · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_abea9d0c
Similarly, this model surpasses PaLM- 540B everywhere but on BoolQ and WinoGrande.

Common Sense Reasoning · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5ae73d52
LLaMA-13B model also outperforms GPT-3 on most benchmarks despite being 10× smaller.

Common Sense Reasoning · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_40d48ce2
On both benchmarks, LLaMA-65B achieve state-of-the-arts performance in the zero-shot and few-shot settings.

Closed-book Question Answering · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d6008731
On these benchmarks, LLaMA-65B is competitive with PaLM-540B, and, LLaMA-13B outperforms GPT-3 by a few percents.

Reading Comprehension · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5283d79b
On GSM8k, we observe that LLaMA- 65B outperforms Minerva-62B, although it has not been fine-tuned on mathematical data.

Mathematical reasoning · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7795710e
LLaMA-65B outperforms Minerva 62B on GSM8k, although it has not been fine-tuned on mathematical data. docstring.

Code generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ad042885
As show in Table 8, for a similar number of parameters, LLaMA outperforms other general models such as LaMDA and PaLM, which are not trained or finetuned specifically for code.

Code generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6389ed60
LLaMA with 13B parameters and more outperforms LaMDA 137B on both HumanEval and MBPP.

Code generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_28916187
LLaMA 65B also outperforms PaLM 62B, even when it is trained longer.

Code generation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c2b51e00
This large quantity of books used by Gopher, Chinchilla and PaLM may also explain why Gopher outperforms GPT-3 on this benchmark, while it is comparable on other benchmarks.

Massive Multitask Language · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_decbc446
On most benchmarks, the performance improves steadily, and correlates with the training perplexity of the model (see Figure 1).

Evolution of performance during training · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b7c2f4f2
On WinoGrande, the performance does not correlate as well with training perplexity: the LLaMA-33B and LLaMA-65B have similar performance during the training.

Evolution of performance during training · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9fb5e22c
In this section, we show that briefly finetuning on instructions data rapidly leads to improvements on MMLU.

Instruction Finetuning · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e876ef95
Although the non-finetuned version of LLaMA-65B is already able to follow basic instructions, we observe that a very small amount of finetuning improves the performance on MMLU, and further improves the ability of the model to follow instructions.

Instruction Finetuning · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_86e8b0b0
In this paper, we presented a series of language models that are released openly, and competitive with state-of-the-art foundation models.

Conclusion · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_412a716b
Unlike previous studies, we show that it is possible to achieve state-of-the-art performance by training exclusively on publicly available data, without resorting to proprietary datasets.

Conclusion · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1fa366b3
Finally, we plan to release larger models trained on larger pretraining corpora in the future, since we have seen a constant improvement in performance as we were scaling.

Conclusion · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2302.13971) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.