MearcAll papers

arXiv 2005.14165 · cs.CL

Language Models are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and 25 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_368f0e0e
Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art finetuning approaches.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_84fa3734
GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8e091a29
4 Figure 1.3: Aggregate performance for all 42 accuracy-denominated benchmarks While zero-shot performance improves steadily with model size, few-shot performance increases more rapidly, demonstrating that larger models are more proficient … (excerpt)

Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ad2d7a98
Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the the few-shot setting is sometimes competitive with or even occasionally surpasses state-of-the-art (despite … (excerpt)

Introduction · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cb2d294c
For example, GPT-3 achieves 81.5 F1 on CoQA in the zero-shot setting, 84.0 F1 on CoQA in the one-shot setting, 85.0 F1 in the few-shot setting.

Introduction · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e6aec785
Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, the last of which is state-of-the-art relative to … (excerpt)

Introduction · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_14b34a2e
We especially highlight the few-shot results as many of them are only slightly behind state-of-the-art fine-tuned models.

Approach · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_82f733bb
Therefore, we took 3 steps to improve the average quality of our datasets: (1) we downloaded and filtered a version of CommonCrawl based on similarity to a range of high-quality … (excerpt)

Training Dataset · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7552dccd
Our largest model sets a new SOTA on PTB by a substantial margin of 15 points, achieving a perplexity of 20.50.

Language Modeling · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0f50f4cc
We find that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of 8% over the previous state of the art.

LAMBADA · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_17b28c0a
Alice went to visit her friend .→ Bob George bought some baseball equipment, a ball, a glove, and a .→ When presented with examples formatted this way, GPT-3 achieves 86.4% … (excerpt)

LAMBADA · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_134309f5
While this setting decreases the performance of the smallest model by almost 20%, for GPT-3 it improves accuracy by 10%.

LAMBADA · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_256c5748
The examples were adversarially mined to be difficult for language models while remaining easy for humans (who achieve 95.6% accuracy).

HellaSwag · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a828adb3
Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot setting (withK = 70).

StoryCloze · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1dbcf62f
This is still 4.1% lower than the fine-tuned SOTA using a BERT based model [ LDL19] but improves over previous zero-shot results by roughly 10%.

StoryCloze · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_20ed1814
On TriviaQA, we achieve 64.3% in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting.

Closed Book Question Answering · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e4e980c0
The zero-shot result already outperforms the fine-tuned T5-11B by 14.2%, and also outperforms a version with Q&A tailored span prediction during pre-training by 3.8%.

Closed Book Question Answering · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_83956d49
GPT-3’s few-shot result further improves performance another 3.2% beyond this.

Closed Book Question Answering · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ac06f4cf
On WebQuestions (WebQs), GPT-3 achieves 14.4% in the zero-shot setting, 25.3% in the one-shot setting, and 41.5% in the few-shot setting.

Closed Book Question Answering · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_af102d70
GPT-3 in the few-shot setting approaches the performance of state-of-the-art fine-tuned models.

Closed Book Question Answering · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6497db26
Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to few-shot (and indeed its zero-shot and one-shot performance are poor), perhaps suggesting that the WebQs questions 13 … (excerpt)

Closed Book Question Answering · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_db7149c5
On Natural Questions (NQs) GPT-3 achieves 14.6% in the zero-shot setting, 23.0% in the one-shot setting, and 29.9% in the few-shot setting, compared to 36.6% for fine-tuned T5 11B+SSM.

Closed Book Question Answering · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7c8f8b06
Since we increase the capacity by over two orders of magnitude from GPT-2 to GPT-3, we also expand the scope of the training dataset to include more representation of other … (excerpt)

Translation · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7a3444c8
GPT-3 in the full few-shot setting further improves another 4 BLEU resulting in similar average performance to prior unsupervised NMT work.

Translation · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ae4177a7
For the three input languages studied, GPT-3 significantly outperforms prior unsupervised NMT work when translating into English but underperforms when translating in the other direction.

Translation · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_04976139
Performance on En-Ro is a noticeable outlier at over 10 BLEU worse than prior unsupervised NMT work.

Translation · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ddb04f53
On Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings, showing no clear in-context learning but in all cases achieving strong results just a few … (excerpt)

Winograd-Style Tasks · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5529fdb1
On the more difficult Winogrande dataset, we do find gains to in-context learning: GPT-3 achieves 70.2% in the zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot … (excerpt)

Winograd-Style Tasks · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_89c82318
GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot (the last measured on PIQA’s test server).

Common Sense Reasoning · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_732b02ef
PIQA shows relatively shallow scaling with model size and is still over 10% worse than human performance, but GPT-3’s few-shot and even zero-shot result outperform the current state-of-the-art.

Common Sense Reasoning · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ec72ac47
On the “Challenge” version of the dataset which has been filtered to questions which simple statistical or information retrieval methods are unable to correctly answer, GPT-3 achieves 51.4% accuracy in … (excerpt)

Common Sense Reasoning · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_65197f9b
However, both of these results are still much worse than the overall SOTAs achieved by the UnifiedQA which exceeds GPT-3’s few-shot results by 27% on the challenge set and 22% … (excerpt)

Common Sense Reasoning · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_04124b0a
Overall, in-context learning with GPT-3 shows mixed results on commonsense reasoning tasks, with only small and inconsistent gains observed in the one and few-shot learning settings for both PIQA and … (excerpt)

Common Sense Reasoning · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_66783cfd
This allows it to slightly outperform the best fine-tuned result in the original paper.

Reading Comprehension · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_21a13980
GPT-3 175B achieves 85 F1 in the few-shot setting, only a few points behind measured human performance and state-of-the-art fine-tuned models.

SuperGLUE · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2e42c9dc
On COPA and ReCoRD GPT-3 achieves near-SOTA performance in the one-shot and few-shot settings, with COPA falling only a couple points short and achieving second place on the leaderboard, where … (excerpt)

SuperGLUE · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3bfc8cd9
On WSC, performance is still relatively strong, achieving 80.1% in the few-shot setting (note that GPT-3 achieves 88.6% on the original Winograd dataset as described in Section 3.4).

SuperGLUE · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4f0268d4
Despite these weaknesses, GPT-3 still outperforms a fine-tuned BERT-large on four of eight tasks and on two tasks GPT-3 is close to the state-of-the-art held by a fine-tuned 11 billion … (excerpt)

SuperGLUE · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e0fe93c6
Finally, we note that the few-shot SuperGLUE score steadily improves with both model size and with number of examples in the context showing increasing benefits from in-context learning (Figure 3.8).

SuperGLUE · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cc0508d9
When sweeping over values ofK, we find that GPT-3 requires less than eight total examples per task to outperform a fine-tuned BERT-Large on overall SuperGLUE score.

SuperGLUE · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2005.14165) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.