clm_368f0e0eHere we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art finetuning approaches.
arXiv 2005.14165 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_368f0e0eHere we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art finetuning approaches.
res_84fa3734GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, … (excerpt)
res_8e091a294 Figure 1.3: Aggregate performance for all 42 accuracy-denominated benchmarks While zero-shot performance improves steadily with model size, few-shot performance increases more rapidly, demonstrating that larger models are more proficient … (excerpt)
res_ad2d7a98Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the the few-shot setting is sometimes competitive with or even occasionally surpasses state-of-the-art (despite … (excerpt)
res_cb2d294cFor example, GPT-3 achieves 81.5 F1 on CoQA in the zero-shot setting, 84.0 F1 on CoQA in the one-shot setting, 85.0 F1 in the few-shot setting.
res_e6aec785Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, the last of which is state-of-the-art relative to … (excerpt)
clm_14b34a2eWe especially highlight the few-shot results as many of them are only slightly behind state-of-the-art fine-tuned models.
res_82f733bbTherefore, we took 3 steps to improve the average quality of our datasets: (1) we downloaded and filtered a version of CommonCrawl based on similarity to a range of high-quality … (excerpt)
res_7552dccdOur largest model sets a new SOTA on PTB by a substantial margin of 15 points, achieving a perplexity of 20.50.
res_0f50f4ccWe find that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of 8% over the previous state of the art.
res_17b28c0aAlice went to visit her friend .→ Bob George bought some baseball equipment, a ball, a glove, and a .→ When presented with examples formatted this way, GPT-3 achieves 86.4% … (excerpt)
res_134309f5While this setting decreases the performance of the smallest model by almost 20%, for GPT-3 it improves accuracy by 10%.
res_256c5748The examples were adversarially mined to be difficult for language models while remaining easy for humans (who achieve 95.6% accuracy).
res_a828adb3Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot setting (withK = 70).
res_1dbcf62fThis is still 4.1% lower than the fine-tuned SOTA using a BERT based model [ LDL19] but improves over previous zero-shot results by roughly 10%.
res_20ed1814On TriviaQA, we achieve 64.3% in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting.
res_e4e980c0The zero-shot result already outperforms the fine-tuned T5-11B by 14.2%, and also outperforms a version with Q&A tailored span prediction during pre-training by 3.8%.
res_83956d49GPT-3’s few-shot result further improves performance another 3.2% beyond this.
res_ac06f4cfOn WebQuestions (WebQs), GPT-3 achieves 14.4% in the zero-shot setting, 25.3% in the one-shot setting, and 41.5% in the few-shot setting.
clm_af102d70GPT-3 in the few-shot setting approaches the performance of state-of-the-art fine-tuned models.
res_6497db26Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to few-shot (and indeed its zero-shot and one-shot performance are poor), perhaps suggesting that the WebQs questions 13 … (excerpt)
res_db7149c5On Natural Questions (NQs) GPT-3 achieves 14.6% in the zero-shot setting, 23.0% in the one-shot setting, and 29.9% in the few-shot setting, compared to 36.6% for fine-tuned T5 11B+SSM.
res_7c8f8b06Since we increase the capacity by over two orders of magnitude from GPT-2 to GPT-3, we also expand the scope of the training dataset to include more representation of other … (excerpt)
res_7a3444c8GPT-3 in the full few-shot setting further improves another 4 BLEU resulting in similar average performance to prior unsupervised NMT work.
res_ae4177a7For the three input languages studied, GPT-3 significantly outperforms prior unsupervised NMT work when translating into English but underperforms when translating in the other direction.
res_04976139Performance on En-Ro is a noticeable outlier at over 10 BLEU worse than prior unsupervised NMT work.
res_ddb04f53On Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings, showing no clear in-context learning but in all cases achieving strong results just a few … (excerpt)
res_5529fdb1On the more difficult Winogrande dataset, we do find gains to in-context learning: GPT-3 achieves 70.2% in the zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot … (excerpt)
res_89c82318GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot (the last measured on PIQA’s test server).
res_732b02efPIQA shows relatively shallow scaling with model size and is still over 10% worse than human performance, but GPT-3’s few-shot and even zero-shot result outperform the current state-of-the-art.
res_ec72ac47On the “Challenge” version of the dataset which has been filtered to questions which simple statistical or information retrieval methods are unable to correctly answer, GPT-3 achieves 51.4% accuracy in … (excerpt)
res_65197f9bHowever, both of these results are still much worse than the overall SOTAs achieved by the UnifiedQA which exceeds GPT-3’s few-shot results by 27% on the challenge set and 22% … (excerpt)
res_04124b0aOverall, in-context learning with GPT-3 shows mixed results on commonsense reasoning tasks, with only small and inconsistent gains observed in the one and few-shot learning settings for both PIQA and … (excerpt)
clm_66783cfdThis allows it to slightly outperform the best fine-tuned result in the original paper.
res_21a13980GPT-3 175B achieves 85 F1 in the few-shot setting, only a few points behind measured human performance and state-of-the-art fine-tuned models.
res_2e42c9dcOn COPA and ReCoRD GPT-3 achieves near-SOTA performance in the one-shot and few-shot settings, with COPA falling only a couple points short and achieving second place on the leaderboard, where … (excerpt)
res_3bfc8cd9On WSC, performance is still relatively strong, achieving 80.1% in the few-shot setting (note that GPT-3 achieves 88.6% on the original Winograd dataset as described in Section 3.4).
res_4f0268d4Despite these weaknesses, GPT-3 still outperforms a fine-tuned BERT-large on four of eight tasks and on two tasks GPT-3 is close to the state-of-the-art held by a fine-tuned 11 billion … (excerpt)
res_e0fe93c6Finally, we note that the few-shot SuperGLUE score steadily improves with both model size and with number of examples in the context showing increasing benefits from in-context learning (Figure 3.8).
res_cc0508d9When sweeping over values ofK, we find that GPT-3 requires less than eight total examples per task to outperform a fine-tuned BERT-Large on overall SuperGLUE score.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2005.14165) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.