MearcAll papers

arXiv 2001.08361 · cs.LG stat.ML

Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, and 4 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedres_475137cf
2 Dataset Size tokens Parameters non-embedding Compute PF-days, non-embedding Test Loss Figure 1 Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute … (excerpt)

Summary · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d213f47
(Section 3) Universality of overfitting: Performance improves predictably as long as we scale upN andD in tandem, but enters a regime of diminishing returns if either N or D is … (excerpt)

Summary · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ca2f2b88
(Section 5) Transfer improves with test performance: When we evaluate models on text with a different distribution than they were trained on, the results are strongly correlated to those on … (excerpt)

Summary · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1b6937e7
(Section 3.2.2) Sample efficiency: Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps (Figure 2) and using fewer data points (Figure … (excerpt)

Summary · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cd5b09ae
(Section 5.1) Taken together, these results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute.

Summary · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_59d6b9f5
3 Larger models require fewer samples to reach the same performance 10 8 6 4 The optimal model size grows smoothly with the loss target and compute budget Line color … (excerpt)

Summary · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d264d15
The power laws αN,α D,α minC specify the degree of performance improvement expected as we scale up N,D, or Cmin; for example, doubling the number of parameters yields a loss … (excerpt)

Summary of Scaling Laws · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_82463a94
To characterize language model scaling we train a wide variety of models, varying a number of factors including: • Model size (ranging in size from 768 to 1.5 billion non-embedding … (excerpt)

Empirical Results and Basic Power Laws · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5949d24c
As shown in Figure 1, we find a steady trend with non-embedding parameter countN, which can be fit to the first term of Equation (1.5), so that L(N )≈ (Nc … (excerpt)

Performance with Non-Embedding Parameter Count N · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8ebb8110
We present power-law relationships between performance and context position Appendix D.5, where increasingly large powers for larger models suggest improved ability to quickly recognize patterns.

Comparing to LSTMs and Universal Transformers · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_444c6d31
Thus for a given value ofC we can scan over all models with variousN to find the model 9 104 105 106 107 108 109 Parameters (non-embedding) 3 4 5 … (excerpt)

Performance with Dataset Size and Compute · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d9ee47ab
The data strongly suggests that sample efficiency improves with model size, and we also illustrate this directly in Figure 19 in the appendix.

Performance with Dataset Size and Compute · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_717c502e
Since apparentlyLmin is much smaller than the values ofL we have achieved, we used a parameterization whereBcrit diverges asL→ 0.

Adjustment for Training at Bcrit(L) · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_997e3c38
However, this result involved training at a fixed batch size B, whereas we know 14 Models between 0.6x and 2.2x the optimal size can be trained with a 20% larger … (excerpt)

Optimal Allocation of the Compute Budget · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dd424438
We see that as compared to the compute plot of Figure 1, the new fit withCmin is somewhat improved.

Optimal Performance and Allocations · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_63592848
For a fixed value of the loss, we predict: Nf Nf′ = ( 1 +f 1 +f′ )1/αN ≈ 2.7 (B.12) Sf Sf′ = ( 1 + 1f 1 + … (excerpt)

Summary of Power Laws · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_09e7d653
(B.17) A 2.2x larger model requires 45% fewer steps at a cost of 20% more training compute.

Suboptimal Model Sizes · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e944f51d
Improvements in these could alter our results, quantitatively or qualitatively. • We used the estimated training compute C≈ 6NBS , which did not include contributions proportional tonctx (see Section 2.1).

Caveats · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6d8ebc1a
Sample efficiency (show here for training far below the critical batch size) improves greatly as well, improving by a factor of almost 100 when comparing the smallest possible model to … (excerpt)

Batch Size · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_62402a3b
We see that models trained on nctx = 1024 show steady improvement with model size on all but the first token.

Context Dependence · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_13740df5
Not only do larger models converge to better performance atT = 1024, but they also improve more quickly at early tokens, suggesting that larger models are more efficient at detecting … (excerpt)

Context Dependence · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fde71501
In the right-hand plot we show how per-token performance varies for a fixed model as a function of the training step.

Context Dependence · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8be91ffc
104 105 106 107 108 109 Parameters (non-embedding) 2 3 4 5 6Test Loss (at convergence) L = (N/8.8 1013) 0.076 L = 0.25log(N/7.1 1012) Figure 23 The trend for … (excerpt)

Learning Rate Schedules and Error Analysis · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_abcfe501
In figure 24 we show that generalization to other data distributions does not depend on network depth when we hold the total parameter count fixed.

Generalization and Architecture · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2001.08361) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.