MearcAll papers

arXiv 2201.11903 · cs.CL cs.AI

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, and 3 more

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedclm_9f08b6f3
In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-ofthought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6970639d
For instance, prompting a PaLM 540B with just eight chain-of-thought exemplars achieves state-of-the-art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7bf7d915
We present empirical evaluations on arithmetic, commonsense, and symbolic reasoning benchmarks, showing that chain-of-thought prompting outperforms standard prompting, sometimes to a striking degree.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bd886dc3
On the other hand, for SingleOp, the easiest subset of MAWPS which only requires a single step to solve, performance improvements were either negative or very small (see Appendix Table 3).

Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d70d661c
Figure 4 shows how PaLM 540B uses chain-ofthought prompting to achieve new state of the art on GSM8K, SV AMP, and MAWPS (though note that standard prompting already passed the prior best for SV AMP).

Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2b23ab32
On the other two datasets, AQuA and ASDiv, PaLM with chain-of-thought prompting reaches within 2% of the state of the art (Appendix Table 2).

Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1d7d5fb6
To better understand why chain-of-thought prompting works, we manually examined modelgenerated chains of thought by LaMDA 137B for GSM8K.

Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_de7f152a
To provide a small insight into why scaling improves chain-of-thought reasoning ability, we performed a similar analysis of errors made by PaLM 62B and whether those errors were fixed by scaling to PaLM 540B.

Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9849565f
Figure 5 shows that equation only prompting does not help much for GSM8K, which implies that the semantics of the questions in GSM8K are too challenging to directly translate into an equation without the natural language reasoning steps in chain of thought.

Ablation Study · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_052c5e15
For datasets of one-step or two-step problems, however, we find that equation only prompting does improve performance, since the equation can be easily derived from the question (see Appendix Table 6).

Ablation Study · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_201cc7e6
2 Figure 6 shows that these prompts performed comparably with our manually written exemplars, also substantially outperforming standard prompting.

Robustness of Chain of Thought · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dbe62e3c
For all tasks, scaling up model size improved the performance of standard prompting; chain-of-thought prompting led to further gains, with improvements appearing to be largest for PaLM 540B.

Commonsense Reasoning · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_228464f2
With chain-of-thought prompting, PaLM 540B achieved strong performance relative to baselines, outperforming the prior state of the art on StrategyQA (75.6% vs 69.4%) and outperforming an unaided sports enthusiast on sports understanding (95.4% vs 84%).

Commonsense Reasoning · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_676f5d73
We show that chain-ofthought prompting not only enables language models to perform symbolic reasoning tasks that are challenging in the standard prompting setting, but also facilitates length generalization to inference-time inputs longer than those seen in the few-shot exemplars.

Symbolic Reasoning · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d3f67c9
We first saw that chain-of-thought prompting improves performance by a large margin on arithmetic reasoning, yielding improvements that are much stronger than ablations and robust to different annotators, exemplars, and language models (Section 3).

Discussion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_69815c49
These chain of thought annotations also outperformed the baseline by a large margin for all four arithmetic datasets (Table 6), indicating that chain of thought is not dependent on a particular set of annotators. • Different exemplars.

Frequently Asked Questions · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_badabb0d
We also found in preliminary experiments that further increasing the number of exemplars in standard prompting did not lead to significant gains (e.g., increasing from 8 to 16 exemplars did not improve the performance of standard prompting enough to catch up with chain-of-thought prompting). • Different language models.

Frequently Asked Questions · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c29a5a95
We find that with the same prompts, chain-of-thought prompting improves performance across all three models (LaMDA, GPT-3, and PaLM) for all datasets except CSQA and StrategyQA for GPT-3 (Table 1, Table 4, Table 5).

Frequently Asked Questions · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_60eb7f52
Though most chain of thought annotations outperform standard prompting, there is large variation in many cases.

Frequently Asked Questions · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b74544f0
The performance gain is small for the subsets of MAWPS that only require one or two steps (SingleOP, SingleEq, and AddSub), for which PaLM 540B already achieves performance of 90% or higher (and it is also generally true that there is less headroom for improvement when performance is already strong).

Frequently Asked Questions · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_aca59822
Chain of thought generally outperforms ablations by a large amount. “Equation only” performs in between standard prompting and chain of thought prompting, as it allows for intermediate reasoning steps via equations but does not leverage natural language.

OOD: 3 OOD: 4 2 OOD: 3 OOD: 4 · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e24188f4
Chain of thought prompting has variance (as expected) when used with prompts written by different annotators or when using other exemplars, but still outperforms standard prompting by a large margin.

OOD: 3 OOD: 4 2 OOD: 3 OOD: 4 · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_08fe750f
Chain of thought generally outperforms ablations by a large amount.

OOD: 3 OOD: 4 2 OOD: 3 OOD: 4 · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8eb818d6
Whereas work in this direction is often task-specific and uses finetuning, we show that chain-of-thought prompting works for a broad range of tasks without any finetuning.

Numeric and logical reasoning · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6081bbdc
As the experiments in the main paper use a fixed number of few-shot exemplars (8; as constrained by the input length of 1024 tokens), we verify that the chain-of-thought prompting is robust to various numbers of few-shot exemplars.

Additional Robustness Analysis · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_702df4f7
We run experiments for LaMDA 137B, comparing chain-of-thought prompting with standard prompting for the five datasets where standard prompting had a mostly flat scaling curve (the largest model did not achieve high performance).

Additional Robustness Analysis · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1f09915c
As shown in Figure 11, the improvement of chain-of-thought prompting over standard prompting remains robust to varying the number of few-shot exemplars in the prompt.

Additional Robustness Analysis · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2201.11903) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.