clm_fca86300We propose a new test to measure a text model’s multitask accuracy.
arXiv 2009.03300 · cs.CY cs.AI cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_fca86300We propose a new test to measure a text model’s multitask accuracy.
res_0e5e2fc4We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average.
res_25d7e66fHowever, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy.
clm_aed50bd9To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce a new benchmark for assessing models across a diverse … (excerpt)
res_2681e72cOn previous benchmarks, smaller models start well above random chance levels and exhibit more continuous improvements with model size increases, but on our test, GPT-3 moves beyond random chance with … (excerpt)
clm_eeeeb4a5Our results indicate that while recent advances have been impressive, state-of-the-art models still struggle at learning and applying knowledge from pretraining.
res_a0b4a26cIn contrast, we find that the X-Large 175 billion parameter GPT-3 model performs substantially better than random, with an accuracy of 43.9%.
res_ff4c2fdbWhile the smaller models have around 25% zero-shot accuracy, Figure 10 in Appendix A shows that the largest GPT-3 model has a much higher zero-shot accuracy of about 37.7%.
res_d6f370cdGPT-3 does better on College Medicine ( 47.4%) and College Mathematics (35.0%) than calculation-heavy Elementary Mathematics (29.9%).
clm_cf4e4ffaWe show the results in Figure 8, which demonstrates that GPT-3 is uncalibrated.
clm_c4bcd83aWe found that it has recently become possible for models to make meaningful progress on the test, but that state-of-the-art models have lopsided performance and rarely excel at any individual … (excerpt)
clm_74dfe3e3Our expansive test can help researchers pinpoint important shortcomings of models, making it easier to gain a clearer picture of state-of-the-art capabilities.
res_14e1bd6eNotably, zero-shot performance is only somewhat lower than 5-shot accuracy.
res_49a50984However, in Figure 13, we see that accuracy and question entropy are not positively correlated, suggesting that the test’s low-entropy questions do not correspond to memorized (and thereby correctly predicted) … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2009.03300) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.