clm_57ec884bWe train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets.
arXiv 2302.13971 · cs.CL
Read on arXiv · CC BY 4.0
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
clm_57ec884bWe train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets.
res_1b3b56c4In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA- 65B is competitive with the best models, Chinchilla-70B and PaLM-540B.
res_7f6eec9dFor instance, although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens.
res_9ceb35b4For instance, LLaMA-13B outperforms GPT-3 on most benchmarks, despite being 10× smaller.
res_ce8e39a72.4 Efficient implementation We make several optimizations to improve the training speed of our models.
res_2c9f3d82First, LLaMA-65B outperforms Chinchilla-70B on all reported benchmarks but BoolQ.
res_abea9d0cSimilarly, this model surpasses PaLM- 540B everywhere but on BoolQ and WinoGrande.
res_5ae73d52LLaMA-13B model also outperforms GPT-3 on most benchmarks despite being 10× smaller.
res_40d48ce2On both benchmarks, LLaMA-65B achieve state-of-the-arts performance in the zero-shot and few-shot settings.
res_d6008731On these benchmarks, LLaMA-65B is competitive with PaLM-540B, and, LLaMA-13B outperforms GPT-3 by a few percents.
res_5283d79bOn GSM8k, we observe that LLaMA- 65B outperforms Minerva-62B, although it has not been fine-tuned on mathematical data.
res_7795710eLLaMA-65B outperforms Minerva 62B on GSM8k, although it has not been fine-tuned on mathematical data. docstring.
res_ad042885As show in Table 8, for a similar number of parameters, LLaMA outperforms other general models such as LaMDA and PaLM, which are not trained or finetuned specifically for code.
res_6389ed60LLaMA with 13B parameters and more outperforms LaMDA 137B on both HumanEval and MBPP.
res_28916187LLaMA 65B also outperforms PaLM 62B, even when it is trained longer.
res_c2b51e00This large quantity of books used by Gopher, Chinchilla and PaLM may also explain why Gopher outperforms GPT-3 on this benchmark, while it is comparable on other benchmarks.
res_decbc446On most benchmarks, the performance improves steadily, and correlates with the training perplexity of the model (see Figure 1).
res_b7c2f4f2On WinoGrande, the performance does not correlate as well with training perplexity: the LLaMA-33B and LLaMA-65B have similar performance during the training.
clm_9fb5e22cIn this section, we show that briefly finetuning on instructions data rapidly leads to improvements on MMLU.
res_e876ef95Although the non-finetuned version of LLaMA-65B is already able to follow basic instructions, we observe that a very small amount of finetuning improves the performance on MMLU, and further improves the ability of the model to follow instructions.
clm_86e8b0b0In this paper, we presented a series of language models that are released openly, and competitive with state-of-the-art foundation models.
clm_412a716bUnlike previous studies, we show that it is possible to achieve state-of-the-art performance by training exclusively on publicly available data, without resorting to proprietary datasets.
clm_1fa366b3Finally, we plan to release larger models trained on larger pretraining corpora in the future, since we have seen a constant improvement in performance as we were scaling.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2302.13971) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.