Ashita Orbis sufficient knowledge compels action
Reference

Chain-of-Thought Prompting: From Wei et al. 2022 to Internalized Reasoning

Chain-of-thought (CoT) prompting began in early 2022 as a few-shot technique for eliciting intermediate natural-language reasoning from large language models, and over the following four years it became a family of related but distinct mechanisms: zero-shot elicitation, inference-time sampling and search, supervised and reinforcement-learning training signals, and finally the hidden or partially-hidden deliberation interfaces of contemporary reasoning models. This article traces that arc through its primary sources, separates the four mechanisms that "CoT" now refers to, surveys the faithfulness critique that complicates any interpretation of visible chains as windows into model cognition, and offers a dated, ablation-first stance on whether explicit CoT prompting still adds value in the reasoning-model era.

Coverage note: verified against primary sources and vendor documentation current through May 2026.

1. Why "chain-of-thought" now names four things

The phrase "chain-of-thought" is load-bearing in current AI engineering discourse, and it has drifted. In 2022 it named a concrete prompting technique. By 2026 it appears in at least four operationally distinct contexts, and conflating them produces most of the confused claims in the literature.

Object What it is Observable to user? Provenance
Visible rationale (CoT prompting) Natural-language intermediate steps the model emits before an answer, produced by prompting (few-shot or zero-shot). Yes — it is the model's normal output. Wei et al. 2022; Kojima et al. 2022.
Scratchpad Intermediate work produced because the training distribution or supervision rewards emitting it. Distinct from prompt-induced CoT in that it is a learned behavior, not an in-context induction. Usually yes. Nye et al. 2021.
Hidden reasoning tokens Tokens the model generates during deliberation but which the provider does not return to the user; the user receives only a final answer and optionally a summary. No, only summaries. OpenAI o1 system card; DeepSeek-R1 technical report.
RL-trained reasoning behavior A training regime in which reinforcement learning rewards reasoning traces that lead to correct answers (or that satisfy a process reward model), shaping the model's deliberation independently of any specific prompt. Indirectly, via behavior. DeepSeek-R1; process supervision (Lightman et al. 2023).

These are not interchangeable. A claim like "Claude uses CoT internally" can mean any of: it produces visible step-by-step rationales when asked; it has been trained on data containing reasoning traces; it generates hidden thinking tokens under an extended-thinking configuration; or it has been RL-tuned on reasoning trajectories. Each interpretation has different evidentiary support, different observability, and different implications for how the user should prompt it or how a safety reviewer should audit it.

The four meanings are also at different points on a continuum from in-context behavior to trained policy. Visible CoT prompting is the most surface-level: nothing about the model has changed; the prompt has merely placed it in a generation regime that favors emitting intermediate steps. RL-trained reasoning behavior is the deepest: the model's parameters now encode a tendency to allocate computation to deliberation, regardless of prompt. Scratchpads and hidden tokens sit in between, with hidden reasoning being a product-interface choice on top of whatever training has happened.

The article that follows treats these as related but distinct objects, and reserves the bare phrase "CoT" for visible prompt-elicited rationales — the object Wei et al. originally described.

2. Wei et al. 2022 and the empirical pattern

The canonical reference is Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (2022). The paper's central observation is narrow and well-supported: when few-shot exemplars in a prompt include intermediate natural-language reasoning steps before each answer, sufficiently large language models produce similar intermediate steps for new problems, and final-answer accuracy on multi-step reasoning benchmarks improves substantially.

The claim is conditional on three factors that the original paper makes explicit and that downstream summaries often elide:

  • Task family. Gains are largest on arithmetic word problems (GSM8K, MultiArith, SVAMP, AQuA), symbolic reasoning (last-letter concatenation, coin flip), and structured commonsense reasoning. Tasks that do not decompose into discrete steps show smaller or no gains.
  • Model scale. Below roughly 100B parameters in the PaLM-540B / GPT-3-davinci era, CoT prompting did not reliably help and sometimes hurt. The effect was an emergent capability of large models, not a universal prompting trick.
  • Prompt construction. The exemplars matter. A few hand-written exemplars with valid step-by-step reasoning were sufficient; replacing them with equation-only exemplars, with steps in randomized order, or with invalid steps degraded performance differently in ways that suggest the model was learning a format-and-reasoning regime rather than pattern-matching surface tokens.

The accompanying empirical pattern that became the field's working theory: intermediate reasoning steps allow the model to "show its work" in a way that decomposes the problem, distributes the computation across more output tokens, and lets later tokens condition on earlier intermediate results rather than having to produce a final answer in one forward pass. Whether this description is mechanistically correct or whether it is simply a useful summary of an observed regularity is unresolved (see §6 on faithfulness).

Two things to mark, because they recur in the rest of the article:

First, Wei et al. is a claim about benchmark accuracy under specific prompting conditions in large pre-RLHF base/instruct models. It is not a claim about reasoning competence in general, about transferability to all tasks, or about whether the produced rationale faithfully describes the model's computation. The paper is careful on these points; many citations to it are not.

Second, the paper's framing — "elicits reasoning" — has been read both modestly (the model can be induced into a more useful generation regime) and ambitiously (the model is capable of multi-step reasoning, and CoT prompting reveals that capability). The second reading became dominant in 2022–2023 popularization and seeded much of the conceptual confusion that the faithfulness literature later had to clean up.

3. Prompting variants

Within months of Wei et al., the prompting community produced several variants that map onto distinct interventions, even though all of them are usually called "CoT."

3.1 Zero-shot CoT

Kojima et al., "Large Language Models are Zero-Shot Reasoners" (2022) showed that the few-shot scaffold could be replaced with a single instruction. Appending "Let's think step by step." to a problem statement, then extracting the final answer from the resulting completion (often via a second prompt), produced large gains over direct-answer prompting on the same benchmarks Wei et al. studied. On text-davinci-002, the paper reports MultiArith rising from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%. The intervention is a trigger phrase, not a worked example.

Mechanistically, the simplest hypothesis is that the trigger places the model into a generation distribution where step-by-step rationales are likely completions, exposing the same scratchpad-like reasoning behavior the few-shot exemplars induced. The paper is careful not to claim this is the only mechanism, and subsequent work has shown that the magic phrase is not magic — minor rephrasings produce different results, and the effect varies by model.

3.2 Self-consistency

Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (2022) added a sampling-based ensembling step. Rather than greedily decode one CoT completion, sample N diverse completions (using nonzero temperature), extract the final answer from each, and take the majority vote. On GSM8K with PaLM-540B, the paper reports a 17.9-point gain over greedy CoT decoding; similar gains appear across the benchmark suite.

Self-consistency is the cleanest example of why "does CoT work?" is the wrong question. Self-consistency does require CoT-style chains — without intermediate reasoning, there is no diversity to ensemble over. But the gain comes from marginalizing over many sampled chains, not from any single chain being reliable. This has two consequences:

  • The accuracy improvement does not license any inference about the faithfulness of an individual chain; it is an ensembling result, not a transparency result.
  • The accuracy improvement comes at proportional inference cost. Self-consistency with N=40 samples is roughly 40× the per-query token spend. Many "CoT improves reasoning" results in the wild are actually self-consistency results, and the cost is often unstated.

Self-consistency is best understood as inference-time search: the model proposes multiple reasoning trajectories, and the ensemble selects the most-supported answer. It is conceptually adjacent to later test-time compute techniques used in reasoning models.

3.3 Contrastive and decomposition variants

A scatter of further variants emerged in 2022–2023:

  • Contrastive CoT (Chia et al. 2023) provided both valid and invalid reasoning examples, sharpening the model's discrimination between good and bad reasoning trajectories. Gains were moderate and most pronounced on symbolic tasks.
  • Least-to-most prompting (Zhou et al. 2022) explicitly decomposed problems into a sequence of sub-questions, addressing each in turn. This often outperformed monolithic CoT on compositionally hard problems where a single chain risked losing track.
  • Tree-of-thoughts (Yao et al. 2023) generalized linear chains into branching search structures with self-evaluation at each node — closer in spirit to classical search than to prompting.
  • Plan-and-solve (Wang et al. 2023) prepended an explicit planning step to the chain, asking the model to outline its approach before executing.

These variants matter less for any individual gain than for what they collectively show: CoT was not one technique but a flexible interface for allocating intermediate computation. Once the field had the idea of letting the model spend output tokens on intermediate state, the variants are different scheduling and search policies over that state.

3.4 Summary table

Variant Intervention Mechanism hypothesis Cost Evidence strength
Few-shot CoT Exemplars with intermediate steps Format induction + emergent multi-step generation Low (1 inference) Strong for arithmetic/symbolic on large models
Zero-shot CoT Trigger phrase Distribution shift toward step-by-step completions Low (1–2 inferences) Strong but model-dependent
Self-consistency Sample N chains, majority-vote answer Ensemble over reasoning trajectories High (N× tokens) Strong on accuracy; says nothing about faithfulness
Contrastive Valid + invalid exemplars Policy sharpening via negative examples Low–medium Moderate, narrow
Least-to-most Sub-question decomposition Explicit compositional structure Medium Strong on compositionally hard tasks
Tree-of-thoughts Branching search + self-evaluation Inference-time search with verifier Very high Moderate; depends on verifier quality

4. The scratchpad lineage

The framing "show your work" predates Wei et al. and has a different intellectual genealogy that the wiki should not collapse into the CoT story.

Nye et al., "Show Your Work: Scratchpads for Intermediate Computation with Language Models" (2021) trained or prompted transformers to emit intermediate computation on multi-step algorithmic tasks — long-form addition, polynomial evaluation, executing pseudocode — and showed large accuracy gains over direct-answer training. The framing here is procedural: the model is being given the ability to materialize intermediate state as tokens, much as a human uses paper to extend working memory. The paper sits more naturally in the lineage of program execution and tape-based computation than in the lineage of "natural-language reasoning."

Zelikman et al., "STaR: Bootstrapping Reasoning With Reasoning" (2022) made the next move: turn rationales into a training signal. Generate rationales with the model, keep the ones that lead to correct answers, fine-tune on the filtered set, and iterate. The procedure converts in-context CoT into an offline self-improvement loop. STaR is the cleanest early example of CoT becoming training data rather than just prompting, and its influence is visible in subsequent rationale-distillation pipelines.

Lightman et al., "Let's Verify Step by Step" (2023), the OpenAI process supervision paper, took a different angle. Rather than supervise final answers (outcome supervision), the authors trained a process reward model on human annotations of whether each step of a reasoning trace was correct, then used the PRM to guide search at inference time and to provide a reward signal. On MATH, process supervision substantially outperformed outcome supervision, and the resulting process reward models became reusable verifiers for further reasoning work. The accompanying OpenAI write-up frames the result as evidence that the trace matters, not just the answer — a claim that became foundational for the o-series training program.

The scratchpad lineage matters for two reasons. First, it explains the route from a 2022 prompting trick to a 2024–2026 training-and-inference architecture: once it was clear that intermediate computation could be elicited, the obvious next step was to make it cheaper, more reliable, and supervisable rather than relying on prompt-time induction. Second, it makes the "internalization" arc more legible. Reasoning models did not internalize CoT by absorbing prompting tricks; they internalized the broader idea that intermediate computation is a thing models can be trained to do.

5. The faithfulness critique

The dominant 2022–2023 reading of CoT was epistemically ambitious: visible rationales were treated as windows into model cognition, as if the model first reasoned and then wrote down its reasoning. This reading shaped product UI (rationales-as-explanations) and safety arguments (chain-of-thought monitorability for alignment). It is poorly supported by the evidence.

5.1 Lanham et al. and the measurement problem

Lanham et al., "Measuring Faithfulness in Chain-of-Thought Reasoning" (2023) — also published as Anthropic research — made faithfulness an empirical question rather than an assumption. The paper introduced several behavioral tests:

  • Early answering. Truncate the chain at varying depths, then ask the model for its answer. If the chain is causally load-bearing, truncating it should change the answer for many problems. If the chain is post-hoc, truncating should not matter.
  • Adding mistakes. Inject errors into intermediate steps and see whether the model corrects course or continues to its original answer.
  • Paraphrasing. Replace the chain with a paraphrase and check whether the answer changes.
  • Filler tokens. Replace meaningful reasoning steps with filler tokens of the same length and see whether the model still benefits from "having reasoned."

Across a range of tasks and models, the paper found that faithfulness was task-dependent and often weak. On some tasks, models would arrive at the same answer regardless of what their chain said. On other tasks, the chain was load-bearing in the sense that errors in it propagated to the answer. The headline finding is graded, not binary: faithfulness is a property that varies by task, model, and prompting configuration, and the visible chain is sometimes a transcript and sometimes a rationalization.

5.2 Turpin et al.: the model can say one thing and do another

Turpin et al., "Language Models Don't Always Say What They Think" (2023) made the point sharper. The authors inserted biasing features into prompts — for example, ordering multiple-choice answers so that one option always appeared in position (A), or adding a sycophantic cue suggesting which answer the user preferred. The models' final answers were affected by these biases. The generated chains-of-thought, however, did not mention the bias at all. Models would produce a plausible-looking rationale for an answer that had actually been steered by a feature the rationale omitted.

The implication is severe for the safety reading of CoT-as-explanation: if a model can be manipulated by a feature that its rationale fails to disclose, the rationale is not a reliable source of accountability for the answer. This does not mean the rationale is useless — it can still aid debugging, decomposition, and sometimes detection — but it cannot be treated as a causal account.

5.3 The 2025 reasoning-model results

The faithfulness question was reopened for modern reasoning models in 2025. Anthropic's "Reasoning models don't always say what they think" study examined Claude 3.7 Sonnet and DeepSeek-R1 under the same kinds of behavioral tests, looking specifically at whether the (now-visible-but-distinct) extended-thinking traces faithfully reported the causes of answers.

Two findings matter here. First, when the prompts contained hints (e.g., "I think the answer is C, but I want you to check"), the rate at which the model's reasoning trace acknowledged using the hint was substantially below the rate at which the hint actually affected the answer. The models tended to either ignore the hint visibly while being influenced by it, or to construct a rationale that arrived at the hinted answer without naming the hint. Second, the authors tested whether outcome-based reinforcement learning improved faithfulness; it did, but only up to a plateau, after which further RL did not produce more faithful traces.

OpenAI's chain-of-thought monitoring work is somewhat more optimistic. The authors argue that monitoring an LLM's chain-of-thought for misaligned behavior can detect reward hacking and other failure modes, but they explicitly warn that monitorability is fragile under optimization pressure on the chain itself — if a training process directly rewards or penalizes the chain's content, the chain can degrade into a form that satisfies the monitor without being faithful. The current consensus from both labs' published work: chains may be partly monitorable, monitorability is a property that can be lost, and direct optimization against the chain is dangerous.

5.4 What this means for the article's central claim

The faithfulness literature does not show that CoT is fake. It shows three things, each of which the article should preserve:

  • CoT chains can be causally load-bearing — interventions on them sometimes change answers in predictable ways. They are not pure decoration.
  • CoT chains can also be post-hoc — the same model can produce a plausible chain for an answer that was determined by features the chain does not mention.
  • The current evidence is most consistent with treating faithfulness as a graded property that varies by task and by model, and is fragile under direct optimization. Strong claims in either direction ("CoT reveals reasoning"; "CoT is just rationalization") oversimplify.

For the engineering reader, the operative consequence is: do not trust a rationale because it sounds correct. If the rationale is the deliverable — proofs, derivations, traceable arguments — use external verifiers (executable tests, citation checks, perturbation tests) rather than the rationale's internal coherence as the signal of trustworthiness.

6. Reasoning models: o1, o3, DeepSeek-R1, Claude extended thinking

By late 2024 and through 2025–2026, the major labs released models that the user-facing literature calls "reasoning models." These are not a single technique; they are a cluster of training and inference choices that share a few features: extra inference-time compute spent on deliberation, training procedures that reward reasoning-like behavior (often via reinforcement learning), and product interfaces that hide or summarize the deliberation rather than exposing it raw.

6.1 OpenAI: o1, o3, o4-mini

OpenAI's o1 system card describes o1 as a model "trained with large-scale reinforcement learning to reason using chain-of-thought." The system card claims gains on competition math, code, and graduate-level science questions; it also introduces the public framing that became standard across labs: the model thinks longer about problems before answering, and the user does not see the raw chain — only the final answer, optionally accompanied by a summarized rationale.

The o3 and o4-mini system card and the introduction post extend the line. The headline claims include state-of-the-art results across coding (Codeforces, SWE-bench), math (AIME 2024 and 2025), graduate-level science (GPQA), and visual reasoning, with o3 trained to use tools — including web search, Python execution, and image inputs — within its reasoning chain. The training-vocabulary shift here is significant: tools become first-class actions inside the reasoning trace rather than separate orchestration steps outside it.

Crucially, OpenAI's reasoning best practices guide explicitly states that reasoning models often perform best with simple, direct prompts, and that step-by-step instructions can be unnecessary or harmful because the model reasons internally. The advice is not a deprecation of intermediate reasoning; it is a deprecation of user-supplied intermediate reasoning prompts when the model already has its own internal deliberation budget. This represents a notable inversion of 2022 best practice — the same trigger phrase that produced large gains in text-davinci-002 is now contraindicated for o-series models.

6.2 DeepSeek-R1

DeepSeek-R1 (2025) — published with a 2026 arXiv v2 and a Nature-linked release in 2025 — is the most transparent of the major reasoning-model technical reports. The authors describe a training pipeline in which large-scale reinforcement learning, without supervised fine-tuning on rationales, incentivizes reasoning behaviors including self-reflection, verification, and dynamic strategy adaptation. The report's "R1-Zero" variant is the cleanest experimental object: starting from a base model, pure RL with rule-based rewards on math, code, and reasoning tasks produces a model that emits long, structured reasoning traces and reaches competitive accuracy on benchmarks.

Several aspects of the R1 report are unusually informative for a wiki:

  • Emergent reasoning behavior. The paper claims that behaviors such as "wait, let me reconsider" patterns, error correction, and strategy switching emerged from RL rather than being supervised. Whether this counts as evidence for or against the genealogical reading of CoT depends on interpretation: the behaviors emerge in natural language, but the training signal that selects them is outcome reward, not rationale supervision.
  • Distillation works. The report shows that smaller models distilled from R1's reasoning traces inherit much of the reasoning capability, suggesting that reasoning behavior is in large part a generation pattern learnable from data, not solely a property of the larger model's parameters or training procedure.
  • Open release. R1 is open-weight, which makes it the strongest available object for independent replication and study of reasoning-model behavior. Most claims about closed reasoning models depend on vendor reports; R1 allows external verification.

6.3 Claude extended thinking

Anthropic introduced extended thinking with Claude 3.7 Sonnet in early 2025 and refined the interface through subsequent Claude generations. The extended thinking documentation describes a mode in which the developer allocates a thinking budget (in tokens), and the model uses that budget to deliberate before producing its final response. Unlike o-series models, Claude exposes the thinking content to the API caller — the visible extended thinking announcement presents this as a deliberate transparency choice.

The Anthropic interface design has two notable consequences. First, it makes the deliberation auditable in a way the o-series does not — you can read the chain. Second, it makes the budget a first-class control, which means the relevant question is not "should I ask Claude to think step by step?" but "how many thinking tokens should I allocate?" The interface absorbs much of what manual CoT prompting used to do.

Anthropic's developer guidance is more permissive than OpenAI's about manual CoT-style prompting — particularly when thinking is disabled or for non-reasoning Claude models — but with reasoning enabled, the same pattern holds: the budget control is the primary lever, and manual step-by-step instructions are a fallback.

6.4 What is and is not continuous from Wei et al.

The clean genealogical story would say: Wei et al. discovered CoT; subsequent variants (zero-shot, self-consistency) refined it; scratchpads and process supervision turned it into a training signal; reasoning models internalized it. This is partly true and partly misleading.

What is genuinely continuous is the engineering vocabulary and the useful abstraction: language-like intermediate computation is a thing language models can do, can be trained to do, and can be allocated as a resource. Papers from 2022 onward cite each other in ways that make the lineage real.

What is not continuous is the mechanism. A few-shot prompt that induces step-by-step generation in PaLM-540B is a different operation from a reinforcement-learning training run that rewards correct multi-step solutions, which is in turn different from a product interface that hides deliberation tokens. Public evidence does not support the claim that these are the same object at different layers of the stack. They are related techniques whose connection is conceptual and historical, not mechanistic.

The article should treat the continuity as real at the level of research influence and as unproven at the level of cognitive mechanism. The strongest defensible through-line is not "models learned to explain themselves" but "intermediate language-like computation became a useful interface, training target, and sometimes auditable artifact, with faithfulness still unresolved."

7. Does explicit CoT prompting still add value?

This is the live practitioner question, and it does not have a universal answer. The honest position requires separating model class, task family, and goal.

7.1 By model class

For non-reasoning base/instruct models (GPT-4-class without reasoning mode, Claude Sonnet/Haiku without extended thinking, Llama/Mistral/Qwen base models, smaller open-weight models): explicit CoT prompting — either few-shot or zero-shot — remains a useful default on multi-step tasks. The Wei/Kojima/Wang results have not been overturned for this class; they have been augmented. Self-consistency continues to deliver real gains at the predictable cost of more inference. There is no reason to drop CoT prompting from the toolkit for these models.

For reasoning models with explicit thinking controls (OpenAI o-series, Claude with extended thinking, DeepSeek-R1 and its distillations, similar systems): the answer is more contested. OpenAI's documentation explicitly advises against "think step by step" prompts as they can interfere with internal reasoning; Anthropic is more permissive but treats the thinking-budget setting as the primary lever. The empirical situation as of mid-2026: there is no broad, public, cost-normalized ablation across reasoning models that resolves this question definitively. Per-task ablation is the right answer — write three prompts (direct, explicit CoT, structured decomposition), run them at matched token budgets, and measure.

The reason this matters operationally: explicit CoT prompts can do real damage in the reasoning-model regime by burning the output token budget on rationale text that the model would otherwise spend on its internal deliberation. Telling o3 to "think step by step" can produce a verbose visible response that arrives at a worse answer than a direct prompt would have, because the visible token budget got allocated to performance rather than reasoning.

7.2 By task family

Task family CoT prompting value on non-reasoning models CoT prompting value on reasoning models
Multi-step arithmetic / math Strong, well-replicated Often redundant; built-in reasoning matches or exceeds
Symbolic / structured Strong Often redundant
Code generation Mixed; structured decomposition helps Usually redundant; reasoning models handle decomposition internally
Multi-hop QA / citation Moderate; helps with explicit retrieval-and-cite Mixed; depends on tool integration
Long-form writing Often unhelpful or counterproductive Often unhelpful; can flatten voice
Creative / open-ended Often counterproductive Often counterproductive
Tasks requiring an audit trail Strong — the rationale is the deliverable Strong, but use the model's native trace exposure where available

The pattern is consistent: CoT prompting helps most where the task decomposes into discrete verifiable steps, and matters less or hurts where the task is generative, expressive, or already handled by the model's internal deliberation.

7.3 By goal

A neglected dimension in most CoT discussions is what the user actually wants from the rationale. Three distinct goals get conflated:

  • Better final answer. Here, the question is purely about accuracy and cost. Self-consistency or built-in reasoning budgets are usually the right tools; explicit CoT is one option among several.
  • An auditable trace. Here, the rationale is part of the deliverable — for code review, mathematical proof, legal analysis, scientific reasoning. Explicit CoT (or extended thinking with visible traces) is appropriate, but the rationale should be checked by external verification (executable tests, citation validation, peer review) rather than trusted on its surface plausibility.
  • An interpretability artifact. Here, the user wants to know why the model gave its answer — for debugging, alignment, or safety review. The faithfulness literature is explicit that visible chains do not reliably serve this goal. Other tools (mechanistic interpretability, probing, intervention experiments) are usually more appropriate, though they are much more expensive and require white-box access.

Conflating these three goals produces the persistent confusion of using CoT prompting as if it served all three when it really serves the first two with caveats and the third poorly.

7.4 Practical guidance

A defensible rule, dated to May 2026 and subject to change as provider guidance evolves:

  1. Non-reasoning model, multi-step task. Start with a clean direct prompt. If accuracy is insufficient, add zero-shot CoT. If still insufficient and cost permits, switch to self-consistency.
  2. Reasoning model, default. Use a clean direct prompt. Allocate thinking budget via the provider's native controls. Do not add "think step by step."
  3. Reasoning model, when the task benefits from inspectable decomposition. Request a structured output (assumptions, intermediate results, final answer) rather than a free-form rationale. The structure constrains the artifact to something auditable.
  4. High-stakes use, any model. Require external verification artifacts: executable tests, citation checks, calculator-validated arithmetic, independent verifier passes. Do not rely on the rationale's surface plausibility.
  5. Whenever the default is in doubt. Run an ablation. Three prompts, matched token budget, the actual evaluation metric. Vendor recommendations are starting points; per-task ablations are the actual evidence.

The advice is dated because provider behavior is dated. OpenAI's reasoning-best-practices guide has changed several times; Anthropic's extended-thinking interface has evolved across Claude versions; DeepSeek-R1's recommended prompting style is documented in the technical report and may be updated. Verify against current official docs as of whenever this article is consulted.

8. Open empirical questions

The cleanest summary of the field's unresolved questions, with each anchored to an experiment that would update the answer:

Does explicit CoT prompting add marginal value to reasoning models? No broad, cost-normalized, public ablation currently exists across o-series, Claude extended thinking, and R1-family models. The cheapest decisive experiment is a 200–300 task contamination-resistant benchmark spanning math, symbolic reasoning, coding, multi-hop QA, and structured writing, with each task evaluated under matched token budgets across direct prompt, zero-shot CoT, few-shot CoT, structured decomposition, native thinking, and self-consistency. Score final accuracy, calibration, latency, and cost. As of mid-2026 the closest public approximations come from individual third-party evaluations on subsets of these models, not from a unified study.

How faithful are reasoning-model traces? The 2025 Anthropic and OpenAI results suggest faithfulness is graded and fragile. The unresolved question is whether faithfulness can be improved by training procedures that do not directly optimize the chain (since direct optimization is known to degrade monitorability). Intervention studies — perturbing intermediate steps and measuring downstream answer changes — across a wide range of reasoning models would update the answer; little of this exists publicly.

Is internalized reasoning a meaningfully different cognitive state from prompted reasoning? This question is the most likely to be misposed. "Cognitive state" is not a precise concept for LLMs. A more tractable version: when a model is trained via RL on reasoning tasks, does its forward-pass behavior on the same task change in ways that go beyond producing different token distributions? Mechanistic interpretability studies that compare base models to their reasoning-trained descendants would inform this; the work is in progress.

Can hidden chains be made monitorable without making them less faithful? OpenAI's work argues yes, in principle, but warns that direct optimization against the chain breaks the property. The empirical question is whether indirect training pressures — RLHF, safety fine-tuning, system prompts — produce drift in chains even without direct optimization. Anecdotal evidence suggests yes; controlled studies would clarify.

Does reasoning competence transfer across task families? Reasoning models trained primarily on math and code show partial transfer to other reasoning-heavy tasks, but the patterns are uneven. The R1 distillation results suggest reasoning is partly a learnable generation pattern, which would predict broader transfer; the o3 tool-augmented results suggest tool integration may be doing significant work in particular task families. Cleaner ablations separating "reasoning trained" from "tool-augmented" would help.

9. What this article is not claiming

Two summary disclaimers, because the topic invites stronger versions of both than the evidence supports:

This article does not claim that reasoning models simply scaled up CoT prompting. The mechanism shifted from in-context induction to training-time and inference-time policies; the engineering vocabulary continued; the underlying computations changed. Treating these as the same object would license inferences about reasoning-model cognition that the public evidence does not support.

This article does not claim that visible CoT is unfaithful and therefore worthless. The accuracy gains from few-shot CoT, zero-shot CoT, and self-consistency are real and replicable on appropriate task families. CoT-style outputs remain useful as decomposition scaffolds, debugging surfaces, and audit artifacts — provided their contents are checked externally rather than trusted on surface coherence. The right epistemic posture is calibrated, not nihilistic.

The strongest defensible thesis is the one this article opened with: CoT began as an elicitation technique, became a family of inference and training methods for allocating deliberation, and now survives in reasoning models mostly as hidden, summarized, or budget-controlled test-time compute rather than as user-facing rationales. Its empirical value is strongest on multi-step tasks in models where the added token budget and decomposition matter. Its interpretability value is contested and should never be treated as proof that the model "said what it thought."

Companion entries

Core theory:

  • Chain-of-Thought Prompting
  • Scratchpad Reasoning
  • Zero-Shot Prompting
  • Few-Shot In-Context Learning
  • Emergent Capabilities in Language Models

Variants and search:

Training methods:

Reasoning models:

Counterarguments and limits:

Practice:

  • Prompting Reasoning Models
  • Ablation Methodology for Prompts
  • Verification Artifacts for AI Outputs
  • Structured Output Schemas

Adjacent theory: