MearcAll papers

arXiv 2009.03300 · cs.CY cs.AI cs.CL

Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and 1 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_fca86300
We propose a new test to measure a text model’s multitask accuracy.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0e5e2fc4
We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_25d7e66f
However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_aed50bd9
To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce a new benchmark for assessing models across a diverse … (excerpt)

I NTRODUCTION · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2681e72c
On previous benchmarks, smaller models start well above random chance levels and exhibit more continuous improvements with model size increases, but on our test, GPT-3 moves beyond random chance with … (excerpt)

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_eeeeb4a5
Our results indicate that while recent advances have been impressive, state-of-the-art models still struggle at learning and applying knowledge from pretraining.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a0b4a26c
In contrast, we find that the X-Large 175 billion parameter GPT-3 model performs substantially better than random, with an accuracy of 43.9%.

R ESULTS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ff4c2fdb
While the smaller models have around 25% zero-shot accuracy, Figure 10 in Appendix A shows that the largest GPT-3 model has a much higher zero-shot accuracy of about 37.7%.

R ESULTS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d6f370cd
GPT-3 does better on College Medicine ( 47.4%) and College Mathematics (35.0%) than calculation-heavy Elementary Mathematics (29.9%).

R ESULTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cf4e4ffa
We show the results in Figure 8, which demonstrates that GPT-3 is uncalibrated.

R ESULTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c4bcd83a
We found that it has recently become possible for models to make meaningful progress on the test, but that state-of-the-art models have lopsided performance and rarely excel at any individual … (excerpt)

C ONCLUSION · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_74dfe3e3
Our expansive test can help researchers pinpoint important shortcomings of models, making it easier to gain a clearer picture of state-of-the-art capabilities.

C ONCLUSION · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_14e1bd6e
Notably, zero-shot performance is only somewhat lower than 5-shot accuracy.

F ORMAT SENSITIVITY · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_49a50984
However, in Figure 13, we see that accuracy and question entropy are not positively correlated, suggesting that the test’s low-entropy questions do not correspond to memorized (and thereby correctly predicted) … (excerpt)

E XACT QUESTION AND ANSWER CONTAMINATION · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2009.03300) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.