MearcAll papers

arXiv 2203.15556 · cs.CL cs.LG

Training Compute-Optimal Large Language Models

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, and 16 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_4395f51b
Not only doesChinchilla outperform its much larger counterpart,Gopher, but its reduced model size reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_32ac4394
We fit a parabola to each IsoFLOPs curve to directly estimate at what model size the minimum loss is achieved (Figure 3 (left)).

Estimating the optimal parameter/training tokens allocation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_609e9f23
The full set of hyperparameters used to trainChinchilla are given in Table 4.Chinchilla uses the same model architecture and training setup asGopher with the exception of the differences listed below. … (excerpt)

Model and training details · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e2fda546
8Interestingly, a model trained with AdamW only passes the training performance of a model trained with Adam around 80% of the way through the cosine cycle, though the ending performance … (excerpt)

Model and training details · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b96b23e7
pubmed_abstracts nih_exporter uspto_backgrounds pubmed_central pile_cc bookcorpus2 stackexchange opensubtitles openwebtext2 hackernews dm_mathematics arxiv freelaw books3 philpapers github ubuntu_irc europarl gutenberg_pg_19 0.00 0.02 0.04 0.06 0.08 0.10 Decrease in bpb compared to … (excerpt)

Language modelling · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_263ba6f9
We also include the average prediction for state of the art accuracy in June 2022/2023 made by 73 competitive human forecasters in Steinhardt (2021). tasks for which leakage is less … (excerpt)

Language modelling · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4bbb72e5
Onthisbenchmark, Chinchilla significantly outperformsGopher despite being much smaller, with an average accuracy of 67.6%(improvingupon Gopherby7.6%).

MMLU · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f1fd97ab
Furthermore,Chinchilla achieves greater than 90% accuracy on 4 different individual tasks–high_school_gov_and_politics, international_law, sociology ,and us_foreign_policy.

MMLU · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_13c12e6c
Toourknowledge,noothermodel has achieved greater than 90% accuracy on a subset.

MMLU · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2fcce4ca
Similar to what we observed in MMLU,Chinchilla outperformsGopher on the vast majority of tasks (see Figure 7).

BIG-bench · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0f3ee796
We find thatChinchilla improves the average performance by 10.7%, reaching an accuracy of 65.1% versus 54.4% forGopher.

BIG-bench · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_797421ed
Of the 62 tasks we consider, Chinchillaperformsworsethan Gopherononlyfour— crash_blossom, dark_humor_detection, 11 college_mathematics econometrics moral_scenarios formal_logic medical_genetics machine_learning public_relations global_facts business_ethics electrical_engineering college_computer_science world_religions high_school_us_history high_school_psychology management high_school_computer_science marketing high_school_physics high_school_macroeconomics … (excerpt)

BIG-bench · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ab4a130d
We find thatChinchilla outperforms bothGopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 8.

Common sense · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_504f4eed
12 crash_blossom dark_humor_detection mathematical_induction logical_args general_knowledge_json Human_organs_senses_multiple_choice formal_fallacies_syllogisms_negation known_unknowns navigate sentence_ambiguity moral_permissibility intent_recognition irony_identification entailed_polarity hyperbaton misconceptions evaluating_information_essentiality similarities_abstraction epistemic_reasoning fantasy_reasoning movie_dialog_same_or_different winowhy novel_concepts discourse_marker_prediction strategyqa causal_judgment hindu_knowledge phrase_relatedness alignment_questionnaire … (excerpt)

Common sense · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fa0bc8da
On the unfiltered set,Chinchilla outperforms GPT-3—see Table 9.

Closed-book question answering · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fd75a73b
We see thatChinchilla matches or outperformsGopher and GPT-3 on all tasks.

Gender bias and toxicity · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fcabe1c3
On all but oneChinchilla outperforms the much larger MT-NLG 530B model.

Gender bias and toxicity · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9c9d187a
On Natural Questions, Chinchilla outperforms GPT-3.

Gender bias and toxicity · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4b63891b
When breaking up examples by male/female gender andgotcha/not gotcha, the largest improvement is on femalegotchaexamples (improvement of 10%).

Gender bias and toxicity · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e5bf5a3a
Rae et al. (2021) found that improving language modelling loss by increasing the number of model parameters has only a negligible effect on toxic text generation (unprompted); here we analyze … (excerpt)

Gender bias and toxicity · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_37351633
While the desire to train these mega-models has led to substantial engineering innovation, we hypothesize that the race to train larger and larger models is resulting in models that are … (excerpt)

Discussion & Conclusion · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3a42bed5
We propose three predictive approaches towards optimally setting model size and training duration, based on the outcome of over 400 training runs.

Discussion & Conclusion · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_670a7c0c
We directly test this hypothesis by trainingChinchilla, a 70B parameter model, and show that it outperformsGopher and even larger models on nearly every measured evaluation task.

Discussion & Conclusion · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_adf21e08
Though there has been significant recent work allowing larger and larger models to be trained, our analysis suggests an increased focus on dataset scaling is needed.

Discussion & Conclusion · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_0a09316c
The methods we propose are easy to reproduce in new settings.

Discussion & Conclusion · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_56630014
Decaying by less (5) is clearly worse.

Consistency of scaling results across datasets · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d29b4546
We find that when the cosine cycle overshoots the number of training steps by more than 25%, performance is noticeably degraded—see Figure A1.10 We use Gaussian smoothing with a window … (excerpt)

Approach 1: Fixing model sizes and varying training sequences · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b8a58cb4
Using the Bayes-classifier𝑓★, the expected-risk minimizer𝑓𝑁 and the “single-epoch empirical-risk minimizer” ¯𝑓𝑁”𝐷, we can finally decompose the loss𝐿„𝑁”𝐷” into 𝐿„𝑁”𝐷” , 𝐿„ ¯𝑓𝑁”𝐷” = 𝐿„𝑓★”‚ 𝐿„𝑓𝑁” 𝐿„𝑓★”‚ 𝐿„ ¯𝑓𝑁”𝐷” 𝐿„𝑓𝑁”“ … (excerpt)

Approach 1: Fixing model sizes and varying training sequences · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_15cd3711
We find that our predicted model outperforms the model predicted by Kaplan et al. (2020) as shown in Figure A4.

Predicted compute optimal frontier for all three methods · page 27 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1ffc6051
We find that, independent of the learning rate schedule, AdamW trained models outperform models trained with Adam.

FLOPs computation · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8778fbfc
The setup used forChinchilla (orange) clearly outperforms the setup used to trainGopher (green).

FLOPs computation · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b80fc299
In Table A5 we show the bits-per-byte (bpb) on The Pile (Gao et al., 2020) ofChinchilla, Gopher, and Jurassic-1.Chinchilla outperformsGopher on all subsets.

The Pile · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0b4e1773
Jurassic-1 outperformsChinchilla on 2 subsets— dm_mathematics and ubuntu_irc.

The Pile · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2203.15556) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.