arXiv 2201.11903 · cs.CL cs.AI
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, and 3 more
Read on arXiv · CC BY 4.0
- No reproduction recorded 27
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-ofthought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting.
Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
For instance, prompting a PaLM 540B with just eight chain-of-thought exemplars achieves state-of-the-art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.
Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
We present empirical evaluations on arithmetic, commonsense, and symbolic reasoning benchmarks, showing that chain-of-thought prompting outperforms standard prompting, sometimes to a striking degree.
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
On the other hand, for SingleOp, the easiest subset of MAWPS which only requires a single step to solve, performance improvements were either negative or very small (see Appendix Table 3).
Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
Figure 4 shows how PaLM 540B uses chain-ofthought prompting to achieve new state of the art on GSM8K, SV AMP, and MAWPS (though note that standard prompting already passed the prior best for SV AMP).
Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
On the other two datasets, AQuA and ASDiv, PaLM with chain-of-thought prompting reaches within 2% of the state of the art (Appendix Table 2).
Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
To better understand why chain-of-thought prompting works, we manually examined modelgenerated chains of thought by LaMDA 137B for GSM8K.
Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
To provide a small insight into why scaling improves chain-of-thought reasoning ability, we performed a similar analysis of errors made by PaLM 62B and whether those errors were fixed by scaling to PaLM 540B.
Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
Figure 5 shows that equation only prompting does not help much for GSM8K, which implies that the semantics of the questions in GSM8K are too challenging to directly translate into an equation without the natural language reasoning steps in chain of thought.
Ablation Study · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
For datasets of one-step or two-step problems, however, we find that equation only prompting does improve performance, since the equation can be easily derived from the question (see Appendix Table 6).
Ablation Study · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
2 Figure 6 shows that these prompts performed comparably with our manually written exemplars, also substantially outperforming standard prompting.
Robustness of Chain of Thought · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
For all tasks, scaling up model size improved the performance of standard prompting; chain-of-thought prompting led to further gains, with improvements appearing to be largest for PaLM 540B.
Commonsense Reasoning · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
With chain-of-thought prompting, PaLM 540B achieved strong performance relative to baselines, outperforming the prior state of the art on StrategyQA (75.6% vs 69.4%) and outperforming an unaided sports enthusiast on sports understanding (95.4% vs 84%).
Commonsense Reasoning · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
We show that chain-ofthought prompting not only enables language models to perform symbolic reasoning tasks that are challenging in the standard prompting setting, but also facilitates length generalization to inference-time inputs longer than those seen in the few-shot exemplars.
Symbolic Reasoning · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions
We first saw that chain-of-thought prompting improves performance by a large margin on arithmetic reasoning, yielding improvements that are much stronger than ablations and robust to different annotators, exemplars, and language models (Section 3).
Discussion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions
These chain of thought annotations also outperformed the baseline by a large margin for all four arithmetic datasets (Table 6), indicating that chain of thought is not dependent on a particular set of annotators. • Different exemplars.
Frequently Asked Questions · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions
We also found in preliminary experiments that further increasing the number of exemplars in standard prompting did not lead to significant gains (e.g., increasing from 8 to 16 exemplars did not improve the performance of standard prompting enough to catch up with chain-of-thought prompting). • Different language models.
Frequently Asked Questions · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that with the same prompts, chain-of-thought prompting improves performance across all three models (LaMDA, GPT-3, and PaLM) for all datasets except CSQA and StrategyQA for GPT-3 (Table 1, Table 4, Table 5).
Frequently Asked Questions · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions
Though most chain of thought annotations outperform standard prompting, there is large variation in many cases.
Frequently Asked Questions · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions
The performance gain is small for the subsets of MAWPS that only require one or two steps (SingleOP, SingleEq, and AddSub), for which PaLM 540B already achieves performance of 90% or higher (and it is also generally true that there is less headroom for improvement when performance is already strong).
Frequently Asked Questions · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions
Chain of thought generally outperforms ablations by a large amount. “Equation only” performs in between standard prompting and chain of thought prompting, as it allows for intermediate reasoning steps via equations but does not leverage natural language.
OOD: 3 OOD: 4 2 OOD: 3 OOD: 4 · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions
Chain of thought prompting has variance (as expected) when used with prompts written by different annotators or when using other exemplars, but still outperforms standard prompting by a large margin.
OOD: 3 OOD: 4 2 OOD: 3 OOD: 4 · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions
Chain of thought generally outperforms ablations by a large amount.
OOD: 3 OOD: 4 2 OOD: 3 OOD: 4 · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions
Whereas work in this direction is often task-specific and uses finetuning, we show that chain-of-thought prompting works for a broad range of tasks without any finetuning.
Numeric and logical reasoning · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions
As the experiments in the main paper use a fixed number of few-shot exemplars (8; as constrained by the input length of 1024 tokens), we verify that the chain-of-thought prompting is robust to various numbers of few-shot exemplars.
Additional Robustness Analysis · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions
We run experiments for LaMDA 137B, comparing chain-of-thought prompting with standard prompting for the five datasets where standard prompting had a mostly flat scaling curve (the largest model did not achieve high performance).
Additional Robustness Analysis · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions
As shown in Figure 11, the improvement of chain-of-thought prompting over standard prompting remains robust to varying the number of few-shot exemplars in the prompt.
Additional Robustness Analysis · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions