clm_4395f51bNot only doesChinchilla outperform its much larger counterpart,Gopher, but its reduced model size reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware.
arXiv 2203.15556 · cs.CL cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_4395f51bNot only doesChinchilla outperform its much larger counterpart,Gopher, but its reduced model size reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware.
res_32ac4394We fit a parabola to each IsoFLOPs curve to directly estimate at what model size the minimum loss is achieved (Figure 3 (left)).
res_609e9f23The full set of hyperparameters used to trainChinchilla are given in Table 4.Chinchilla uses the same model architecture and training setup asGopher with the exception of the differences listed below. … (excerpt)
res_e2fda5468Interestingly, a model trained with AdamW only passes the training performance of a model trained with Adam around 80% of the way through the cosine cycle, though the ending performance … (excerpt)
res_b96b23e7pubmed_abstracts nih_exporter uspto_backgrounds pubmed_central pile_cc bookcorpus2 stackexchange opensubtitles openwebtext2 hackernews dm_mathematics arxiv freelaw books3 philpapers github ubuntu_irc europarl gutenberg_pg_19 0.00 0.02 0.04 0.06 0.08 0.10 Decrease in bpb compared to … (excerpt)
res_263ba6f9We also include the average prediction for state of the art accuracy in June 2022/2023 made by 73 competitive human forecasters in Steinhardt (2021). tasks for which leakage is less … (excerpt)
res_4bbb72e5Onthisbenchmark, Chinchilla significantly outperformsGopher despite being much smaller, with an average accuracy of 67.6%(improvingupon Gopherby7.6%).
res_f1fd97abFurthermore,Chinchilla achieves greater than 90% accuracy on 4 different individual tasks–high_school_gov_and_politics, international_law, sociology ,and us_foreign_policy.
res_13c12e6cToourknowledge,noothermodel has achieved greater than 90% accuracy on a subset.
res_2fcce4caSimilar to what we observed in MMLU,Chinchilla outperformsGopher on the vast majority of tasks (see Figure 7).
res_0f3ee796We find thatChinchilla improves the average performance by 10.7%, reaching an accuracy of 65.1% versus 54.4% forGopher.
res_797421edOf the 62 tasks we consider, Chinchillaperformsworsethan Gopherononlyfour— crash_blossom, dark_humor_detection, 11 college_mathematics econometrics moral_scenarios formal_logic medical_genetics machine_learning public_relations global_facts business_ethics electrical_engineering college_computer_science world_religions high_school_us_history high_school_psychology management high_school_computer_science marketing high_school_physics high_school_macroeconomics … (excerpt)
res_ab4a130dWe find thatChinchilla outperforms bothGopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 8.
res_504f4eed12 crash_blossom dark_humor_detection mathematical_induction logical_args general_knowledge_json Human_organs_senses_multiple_choice formal_fallacies_syllogisms_negation known_unknowns navigate sentence_ambiguity moral_permissibility intent_recognition irony_identification entailed_polarity hyperbaton misconceptions evaluating_information_essentiality similarities_abstraction epistemic_reasoning fantasy_reasoning movie_dialog_same_or_different winowhy novel_concepts discourse_marker_prediction strategyqa causal_judgment hindu_knowledge phrase_relatedness alignment_questionnaire … (excerpt)
res_fa0bc8daOn the unfiltered set,Chinchilla outperforms GPT-3—see Table 9.
res_fd75a73bWe see thatChinchilla matches or outperformsGopher and GPT-3 on all tasks.
res_fcabe1c3On all but oneChinchilla outperforms the much larger MT-NLG 530B model.
res_9c9d187aOn Natural Questions, Chinchilla outperforms GPT-3.
res_4b63891bWhen breaking up examples by male/female gender andgotcha/not gotcha, the largest improvement is on femalegotchaexamples (improvement of 10%).
res_e5bf5a3aRae et al. (2021) found that improving language modelling loss by increasing the number of model parameters has only a negligible effect on toxic text generation (unprompted); here we analyze … (excerpt)
clm_37351633While the desire to train these mega-models has led to substantial engineering innovation, we hypothesize that the race to train larger and larger models is resulting in models that are … (excerpt)
clm_3a42bed5We propose three predictive approaches towards optimally setting model size and training duration, based on the outcome of over 400 training runs.
res_670a7c0cWe directly test this hypothesis by trainingChinchilla, a 70B parameter model, and show that it outperformsGopher and even larger models on nearly every measured evaluation task.
clm_adf21e08Though there has been significant recent work allowing larger and larger models to be trained, our analysis suggests an increased focus on dataset scaling is needed.
clm_0a09316cThe methods we propose are easy to reproduce in new settings.
res_56630014Decaying by less (5) is clearly worse.
res_d29b4546We find that when the cosine cycle overshoots the number of training steps by more than 25%, performance is noticeably degraded—see Figure A1.10 We use Gaussian smoothing with a window … (excerpt)
res_b8a58cb4Using the Bayes-classifier𝑓★, the expected-risk minimizer𝑓𝑁 and the “single-epoch empirical-risk minimizer” ¯𝑓𝑁𝐷, we can finally decompose the loss𝐿„𝑁𝐷” into 𝐿„𝑁𝐷” , 𝐿„ ¯𝑓𝑁𝐷” = 𝐿„𝑓★”‚ 𝐿„𝑓𝑁” 𝐿„𝑓★”‚ 𝐿„ ¯𝑓𝑁𝐷” 𝐿„𝑓𝑁” … (excerpt)
clm_15cd3711We find that our predicted model outperforms the model predicted by Kaplan et al. (2020) as shown in Figure A4.
clm_1ffc6051We find that, independent of the learning rate schedule, AdamW trained models outperform models trained with Adam.
clm_8778fbfcThe setup used forChinchilla (orange) clearly outperforms the setup used to trainGopher (green).
res_b80fc299In Table A5 we show the bits-per-byte (bpb) on The Pile (Gao et al., 2020) ofChinchilla, Gopher, and Jurassic-1.Chinchilla outperformsGopher on all subsets.
res_0b4e1773Jurassic-1 outperformsChinchilla on 2 subsets— dm_mathematics and ubuntu_irc.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2203.15556) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.