arXiv 2206.07682 · cs.CL
Emergent Abilities of Large Language Models
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, and 10 more
Read on arXiv · CC BY 4.0
- No reproduction recorded 10
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
For example, later in Figure 4 we will also plot emergence as a function of WikiText103 perplexity (Merity et al., 2016), which happens to closely correlate with training computation for Gopher/ Chinchilla (though this correlation may not hold in the long-run).
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
For GPT-3, Gopher, and Chinchilla, models of∼1022 training FLOPs (∼10B parameters) or smaller do not perform better than guessing on average over all the topics, scaling up to3–5 ·1023 training FLOPs (70B–280B parameters) enables performance to substantially surpass random.
model · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions
As shown in Figure 3A, chain of thought prompting only surpasses standard prompting without intermediate steps when scaled to1023 training FLOPs (∼100B parameters).
NLU task average · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
For example, there are 14 BIG-Bench tasks5 for which LaMDA 137B and GPT-3 175B models perform at near-random, but PaLM 62B in fact achieves above-random performance, despite having fewer model parameters and training FLOPs.
Beyond scaling · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
One key set of results in the emergent sociological shift towards general-purpose models is when scaling enables a few-shot prompted general-purpose model to outperform prior state of the art held by finetuned task-specific models.
Sociological changes · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions
In all of these cases, state-of-the-art performance was achieved by few-shot prompting a language model of unprecendented scale (scaling curves for these examples are shown in Appendix Figure 13).
Sociological changes · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions
This outcome implies that for these scales, the model truly does not get any better at the tasks. • Outcome 2: For the model scales where EM/BLEU/acc is random, cross-entropy loss does improve.
Cross-entropy loss analysis · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions
For all three tasks, notice that while the error rate is nearly 100% for small models (≤1022 FLOPs / ≤27B params), the cross-entropy loss does actually improve for these model sizes.
Generative tasks · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions
The remaining 40 BIG-Bench tasks not included in this chart did not fit into any of the above categories (e.g., too noisy due to very few eval examples, performance not correlated with model scale, etc.).
BIG-Bench task analysis · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions
So the categories that improved the most from 7B to 70B/280B are humanities and social science, whereas STEM (Science, Technology, Engineering, and Mathematics) improved the least.
Further MMLU analysis · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions