res_475137cf2 Dataset Size tokens Parameters non-embedding Compute PF-days, non-embedding Test Loss Figure 1 Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute … (excerpt)
arXiv 2001.08361 · cs.LG stat.ML
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
res_475137cf2 Dataset Size tokens Parameters non-embedding Compute PF-days, non-embedding Test Loss Figure 1 Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute … (excerpt)
res_9d213f47(Section 3) Universality of overfitting: Performance improves predictably as long as we scale upN andD in tandem, but enters a regime of diminishing returns if either N or D is … (excerpt)
res_ca2f2b88(Section 5) Transfer improves with test performance: When we evaluate models on text with a different distribution than they were trained on, the results are strongly correlated to those on … (excerpt)
res_1b6937e7(Section 3.2.2) Sample efficiency: Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps (Figure 2) and using fewer data points (Figure … (excerpt)
res_cd5b09ae(Section 5.1) Taken together, these results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute.
res_59d6b9f53 Larger models require fewer samples to reach the same performance 10 8 6 4 The optimal model size grows smoothly with the loss target and compute budget Line color … (excerpt)
res_9d264d15The power laws αN,α D,α minC specify the degree of performance improvement expected as we scale up N,D, or Cmin; for example, doubling the number of parameters yields a loss … (excerpt)
res_82463a94To characterize language model scaling we train a wide variety of models, varying a number of factors including: • Model size (ranging in size from 768 to 1.5 billion non-embedding … (excerpt)
res_5949d24cAs shown in Figure 1, we find a steady trend with non-embedding parameter countN, which can be fit to the first term of Equation (1.5), so that L(N )≈ (Nc … (excerpt)
clm_8ebb8110We present power-law relationships between performance and context position Appendix D.5, where increasingly large powers for larger models suggest improved ability to quickly recognize patterns.
res_444c6d31Thus for a given value ofC we can scan over all models with variousN to find the model 9 104 105 106 107 108 109 Parameters (non-embedding) 3 4 5 … (excerpt)
res_d9ee47abThe data strongly suggests that sample efficiency improves with model size, and we also illustrate this directly in Figure 19 in the appendix.
res_717c502eSince apparentlyLmin is much smaller than the values ofL we have achieved, we used a parameterization whereBcrit diverges asL→ 0.
res_997e3c38However, this result involved training at a fixed batch size B, whereas we know 14 Models between 0.6x and 2.2x the optimal size can be trained with a 20% larger … (excerpt)
res_dd424438We see that as compared to the compute plot of Figure 1, the new fit withCmin is somewhat improved.
res_63592848For a fixed value of the loss, we predict: Nf Nf′ = ( 1 +f 1 +f′ )1/αN ≈ 2.7 (B.12) Sf Sf′ = ( 1 + 1f 1 + … (excerpt)
res_09e7d653(B.17) A 2.2x larger model requires 45% fewer steps at a cost of 20% more training compute.
res_e944f51dImprovements in these could alter our results, quantitatively or qualitatively. • We used the estimated training compute C≈ 6NBS , which did not include contributions proportional tonctx (see Section 2.1).
res_6d8ebc1aSample efficiency (show here for training far below the critical batch size) improves greatly as well, improving by a factor of almost 100 when comparing the smallest possible model to … (excerpt)
res_62402a3bWe see that models trained on nctx = 1024 show steady improvement with model size on all but the first token.
res_13740df5Not only do larger models converge to better performance atT = 1024, but they also improve more quickly at early tokens, suggesting that larger models are more efficient at detecting … (excerpt)
clm_fde71501In the right-hand plot we show how per-token performance varies for a fixed model as a function of the training step.
res_8be91ffc104 105 106 107 108 109 Parameters (non-embedding) 2 3 4 5 6Test Loss (at convergence) L = (N/8.8 1013) 0.076 L = 0.25log(N/7.1 1012) Figure 23 The trend for … (excerpt)
clm_abcfe501In figure 24 we show that generalization to other data distributions does not depend on network depth when we hold the total parameter count fixed.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2001.08361) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.