MearcAll papers

arXiv 2206.07682 · cs.CL

Emergent Abilities of Large Language Models

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, and 10 more

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedres_e76fd1d0
For example, later in Figure 4 we will also plot emergence as a function of WikiText103 perplexity (Merity et al., 2016), which happens to closely correlate with training computation for Gopher/ Chinchilla (though this correlation may not hold in the long-run).

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bee24776
For GPT-3, Gopher, and Chinchilla, models of∼1022 training FLOPs (∼10B parameters) or smaller do not perform better than guessing on average over all the topics, scaling up to3–5 ·1023 training FLOPs (70B–280B parameters) enables performance to substantially surpass random.

model · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_76173b44
As shown in Figure 3A, chain of thought prompting only surpasses standard prompting without intermediate steps when scaled to1023 training FLOPs (∼100B parameters).

NLU task average · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_992ebdb4
For example, there are 14 BIG-Bench tasks5 for which LaMDA 137B and GPT-3 175B models perform at near-random, but PaLM 62B in fact achieves above-random performance, despite having fewer model parameters and training FLOPs.

Beyond scaling · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_def7b61d
One key set of results in the emergent sociological shift towards general-purpose models is when scaling enables a few-shot prompted general-purpose model to outperform prior state of the art held by finetuned task-specific models.

Sociological changes · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_12c22413
In all of these cases, state-of-the-art performance was achieved by few-shot prompting a language model of unprecendented scale (scaling curves for these examples are shown in Appendix Figure 13).

Sociological changes · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_33c8da09
This outcome implies that for these scales, the model truly does not get any better at the tasks. • Outcome 2: For the model scales where EM/BLEU/acc is random, cross-entropy loss does improve.

Cross-entropy loss analysis · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_50fa1da8
For all three tasks, notice that while the error rate is nearly 100% for small models (≤1022 FLOPs / ≤27B params), the cross-entropy loss does actually improve for these model sizes.

Generative tasks · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_31cabf17
The remaining 40 BIG-Bench tasks not included in this chart did not fit into any of the above categories (e.g., too noisy due to very few eval examples, performance not correlated with model scale, etc.).

BIG-Bench task analysis · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8b601d8a
So the categories that improved the most from 7B to 70B/280B are humanities and social science, whereas STEM (Science, Technology, Engineering, and Mathematics) improved the least.

Further MMLU analysis · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2206.07682) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.