Process Reward Models: Stepwise Supervision for Reasoning

Process reward models (PRMs) score the intermediate steps of a reasoning trace rather than only the final answer. They have strong evidence as verifier and search instruments in mathematical reasoning, but the public literature does not establish that explicit process supervision is necessary for frontier reasoning — recent reasoning-model reports show outcome-only or rule-based reinforcement learning reaching competitive results in verifiable domains without neural PRMs. This article treats the PRM/ORM choice as a design space over label source, granularity, and use mode, surveys the canonical evidence, and names the active disagreement explicitly.

Coverage note: verified through May 10, 2026.

The reward-granularity question

The clean phrasing — "process versus outcome supervision" — is misleading because it suggests two monolithic training methods. The actual decision space has at least three axes that the literature regularly conflates:

Granularity of the reward signal. Does the model receive feedback on the whole trajectory or on individual steps?
Source of the labels. Are step-level judgments produced by human annotators, by automated rollouts from intermediate states, by an LLM-as-judge, by a deterministic rule, or by an implicit objective?
Use mode. Is the reward used to filter SFT data, to rerank candidates at inference, to guide tree or beam search, or to provide a dense reward during reinforcement learning?

Almost every dispute about PRMs that looks like a methodological question on its surface turns out to depend on which combination of axes is being compared. A PRM trained from rollout-derived labels and used to rerank best-of-N is a different artifact from a PRM trained from human step labels and used as a dense RL reward, even when both are called "process reward models." Most of the conflicting claims in the literature dissolve once these axes are tracked separately.

This article preserves that structure. It starts with definitions and the canonical positive case, then walks the failure modes, the automation literature, the evaluation problems, and the live disagreement about whether process supervision is required at the frontier. The conditional thesis defended throughout is: dense process credit is empirically useful when intermediate correctness is meaningful and reliably labelable, and especially when used for verifier-guided search; it has not been shown to be necessary, and outcome or rule-based rewards plus scaled sampling and search are a serious competing approach in verifiable domains.

Outcome reward models: the verifier baseline

The reward-model lineage begins with outcome verifiers. Cobbe et al. 2021 ("Training Verifiers to Solve Math Word Problems") showed that on GSM8K, sampling many candidate solutions from a generator and selecting with a learned verifier trained on final-answer correctness substantially outperformed greedy decoding from the same generator. The verifier scored entire solutions on the binary question "does this reach the correct answer?" and the resulting best-of-N selection scaled well with sample count. This is the canonical outcome reward model (ORM) result, and it established the broader pattern — verifier-guided inference — that frames every later debate.

An ORM has three practical properties that matter for what follows. First, its label source is cheap and unambiguous when ground-truth answers exist: the answer either checks or it does not. Second, it provides a sparse signal — one bit per candidate. Third, it is agnostic about how the answer was produced. A solution that arrives at the correct answer via a flawed chain of reasoning is rewarded identically to one that arrives correctly.

The third property is the seam where PRMs enter. If reasoning fidelity matters — for oversight, for transfer to harder problems, for tool use where mid-trajectory errors compound — an outcome-only signal is structurally insufficient to distinguish those cases.

The Uesato hinge: when does reasoning quality matter?

Uesato et al. 2022 ("Solving math word problems with process- and outcome-based feedback") ran the comparison that the field has been re-running under different names ever since. On GSM8K, they trained reward models with outcome-based supervision (final-answer correctness) and with process-based supervision (human step labels), and they evaluated both as best-of-N verifiers and as RL reward models. Their finding was nuanced: outcome supervision achieved similar final-answer error rates with less labeling effort, but process supervision was important for reducing the rate of trace errors — cases where the model reached the correct answer through invalid intermediate steps.

This is the load-bearing distinction that the later literature too often elides. If the evaluation metric is final-answer accuracy on benchmarks with checkable answers, outcome supervision tends to keep pace. If the evaluation metric includes the validity of the reasoning trace, or if the deployment context demands that wrong-for-the-right-reasons traces be filtered out, process supervision starts to matter independently. Most of the "PRM vs ORM" experimental disagreements in 2023–2026 are visible in retrospect as restagings of Uesato's hinge in different domains and at different scales.

Lightman et al. 2023: the canonical PRM result

Lightman et al. 2023 ("Let's Verify Step by Step") (OpenAI PDF) is the most-cited piece of evidence for process supervision and the load-bearing reference for any article on PRMs. The paper's structure deserves to be read precisely because the results are frequently overgeneralized.

Setup. The authors collected PRM800K, a dataset of approximately 800,000 step-level labels over 75,000 model-generated MATH solutions. Annotators marked each step as positive, negative, or neutral. They trained two reward models from the same base model: an ORM trained on final-answer correctness, and a PRM trained on the step labels. Both reward models were used as best-of-N verifiers — the generator produced N candidate solutions, the reward model scored them, and the top-scoring candidate was taken as the prediction.

Headline numbers. On a representative subset of MATH, with N = 1860 candidates per problem, the PRM solved approximately 78.2% of problems, the ORM solved 72.4%, and majority voting solved 69.6%. The PRM's advantage grew with N — it benefited more from additional samples than the ORM did, suggesting better discrimination among hard cases.

The paper's own caveats. Lightman et al. were explicit about what their experiment did and did not show:

The PRM and ORM training sets were not directly comparable in terms of cost or label structure, so the comparison is not a strict matched-budget result. PRM800K is expensive to collect; the ORM training data is automatically generated by checking final answers.
The experiment evaluated reward models as best-of-N selectors, not as dense rewards in an online RL loop. Claims about PRMs improving RL training do not follow directly from this paper.
The PRM advantage was robust across N but was demonstrated only on MATH-style problems. Transfer to other reasoning domains was not the paper's subject.
The paper explicitly frames process supervision as having alignment-relevant properties (interpretable feedback, ability to inspect rationale) alongside the accuracy gain.

These caveats matter because the result is frequently cited as if it established that PRMs dominate ORMs in general. What it established is much narrower and stronger as a narrow claim: under one specific best-of-N verifier setup on MATH, with very expensive human step labels, a process-supervised reward model selected correct solutions more reliably than an outcome-supervised reward model from the same base. That is a real result; it is not a universal verdict.

The labeling bottleneck

PRM800K is the most ambitious public process-supervision dataset, and it is also a clear demonstration of why human process supervision does not obviously scale. Labeling MATH solutions step-by-step requires annotators with strong mathematical training, the labels are not independent (a step's correctness often depends on earlier steps), and what counts as "a step" is itself a modeling decision rather than a fact about reasoning.

Several specific labeling problems recur:

Step segmentation. Models do not natively produce solutions in discrete labeled steps. Where one step ends and the next begins is a choice that affects both annotation cost and downstream reward shape. Solutions can be parsed by newline, by sentence, by semantic block, or by explicit "step N:" markers, and the choice changes the dataset.
First-error vs full-trace labeling. Some pipelines label every step; others label only up to the first error, treating subsequent steps as undefined (because reasoning after an error may be salvage, conditional on a wrong premise, or simply incoherent). These produce different training signals.
Annotator disagreement. Process labels on intermediate steps in non-trivial mathematics are not perfectly objective. A step can be technically valid but unmotivated, locally correct but globally a dead end, or stylistically unconventional in ways that affect human raters' calibration.
Domain dependence. Step labels for olympiad geometry, code synthesis, formal theorem proving, scientific reasoning, and agentic planning are not interchangeable expertise. Scaling PRM800K to non-math domains requires assembling new annotator pipelines for each.

This is why automated PRM construction became the dominant research direction after 2023. Human process supervision works in the narrow setting where Lightman et al. demonstrated it; whether it generalizes economically to other reasoning domains is a separate question with no positive answer in the public record.

Automated PRM construction

If human step labels are expensive, the obvious substitute is automated step labels — labels derived from the model itself, from outcome checkers, or from other models acting as judges. This is now a substantial subliterature, and reading it carefully matters because the methods share a name but not always a mechanism.

Math-Shepherd: outcome-derived stepwise labels

Wang et al. 2023 ("Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations") introduced the dominant automated approach. For each intermediate step in a candidate solution, the method runs Monte Carlo rollouts from that step to completion and measures the fraction of completions that reach a correct final answer. That fraction becomes the step's pseudo-label: a step from which many completions succeed is labeled positive; a step from which few succeed is labeled negative. The resulting labels are used to train a step verifier that can score new candidates without further rollouts.

Math-Shepherd reported gains in both verification (best-of-N selection) and reinforcement learning (PPO-style training against the learned step verifier) on math benchmarks, with no human step annotation. Subsequent automated process-supervision work — Luo et al. 2024 on Gemma/Gemini, the OmegaPRM-style rollout pipelines, and various open-model implementations — extended this template.

This is also the seam where the field's most consequential interpretive disagreement lives.

The reductionist reading. If step labels are produced by running rollouts and checking final answers, then the supervision signal is ultimately outcome supervision. The Math-Shepherd verifier is a learned compression of "from this state, how likely is the model to reach a correct final answer?" — which is exactly what a value function estimates from outcome rewards. On this reading, automated PRM success does not support the human-process-supervision thesis. It supports the thesis that outcome rewards plus enough rollouts can produce dense heuristics, and that those heuristics can guide search.

The functionalist reading. Even granting that the labels are outcome-derived, the learned object is a step-level value estimate, which is a different artifact from an ORM in use. The Math-Shepherd verifier scores partial trajectories before any final answer exists, changing the search topology — pruning bad branches early, allocating compute to promising prefixes, providing dense feedback during RL. Whether the labels were originally outcome-derived matters less than what the resulting model lets the search procedure do.

Both readings are coherent, and the disagreement is substantive rather than terminological. The reductionist reading correctly notes that "process supervision" in the Lightman sense — humans labeling step correctness — is not what Math-Shepherd does, and so Math-Shepherd's success is not evidence for human process supervision specifically. The functionalist reading correctly notes that "a dense step-level scorer changes inference and training in ways that an outcome scorer cannot," regardless of label provenance. The right way to think about it: automated PRMs are a separate empirical object that should be evaluated on its own terms, neither inherited as further evidence for the Lightman thesis nor dismissed as ORM-in-disguise.

Process advantage verifiers

Setlur et al. 2024 ("Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning") sharpens the distinction. The paper argues that the right object to learn is not "is this step correct?" but "does this step make progress under a different prover policy?" — i.e., an advantage estimate measuring how much the step improves the conditional probability of success relative to a reference policy. The authors report that process advantage verifiers (PAVs) outperform standard PRMs both as test-time search guides and as RL training signals.

PAVs are important to the design space because they make explicit something that was implicit in the earlier automated PRM work: the useful signal for search and RL is progress, not local correctness. A step can be locally valid but make no progress (restating known facts), or locally weird but highly informative (a productive detour). Optimizing for local correctness rewards the first; optimizing for progress rewards the second. This is part of why automated PRMs trained on rollout success rates often outperform PRMs trained on naive correctness labels — the rollout signal is implicitly a progress signal.

Implicit PRMs and outcome-only formulations

A third line of work — sometimes called "implicit PRMs" — derives stepwise reward signals from outcome-only training objectives without ever constructing explicit step labels. Methods in this category include reward shaping from token-level log-probability gaps under DPO-style training, and approaches that derive per-step credit from a learned value function trained purely on final outcomes. The relevant point for the design space: stepwise gradients can sometimes be obtained without stepwise labels, which further weakens the assumption that "process supervision" requires a separate annotation pipeline.

Verifier-guided search

The cleanest empirical wins for PRMs sit in verifier-guided search, where the reward model is used to rerank candidates or to direct a tree search procedure at inference time. This is also the use mode where the design choices interact most cleanly with compute accounting.

In best-of-N selection, the generator produces N candidates and the reward model picks one. Compute scales linearly in N. A PRM that scores each step lets the system reject partial trajectories before completion, which can be cheaper than generating N full candidates and selecting at the end. Lightman et al.'s best-of-1860 result was obtained without truncating partial generations, so it does not exploit this advantage; later work that integrates PRMs into tree search or beam search does.

In tree-based search (MCTS variants, beam search with verifier scoring, lookahead planning), PRMs provide the value estimates that make non-trivial search worthwhile. Outcome-only verifiers cannot guide tree expansion because they require a complete trajectory to score; PRMs can. This is structurally why PRM-style scorers appear in research on reasoning-time search even when the labels are outcome-derived — the search procedure needs a function that takes a partial trajectory and returns a score, and an ORM cannot provide that function.

The relevant counterargument is that the search-time advantage of PRMs over ORMs may not survive compute accounting. If a PRM costs more to train (more annotations, more rollouts, more judge calls) and more to query at inference (one call per step rather than one per candidate), the additional compute could be spent instead on more candidate generations or longer chains of thought from the generator itself. Whether the PRM-guided search wins under matched total compute depends on the ratio of step cost to candidate cost, the value of early pruning, and how reliable the PRM is on the specific problem distribution.

Evaluation: what we are actually measuring

The PRM literature is not just empirically contested; its evaluation methodology is contested. Three lines of work have made the evaluation problems concrete.

ProcessBench and first-error localization

Zheng et al. 2024 ("ProcessBench: Identifying Process Errors in Mathematical Reasoning") constructed a benchmark that evaluates whether a reward model can identify the first erroneous step in a flawed reasoning trace, rather than only ranking complete trajectories by final correctness. This is a sharper test than best-of-N accuracy because it directly probes whether the reward model has learned to distinguish good from bad reasoning at the step level, rather than learning to predict which trajectories happen to reach correct answers.

The ProcessBench results are sobering for the PRM literature. Many existing PRMs that perform well on best-of-N final-answer selection perform poorly on first-error localization, especially on harder problems. This is consistent with the Uesato 2022 finding: PRMs that learn from rollout-derived labels can pick up on surface features predictive of final success without actually identifying step-level errors. Best-of-N performance and first-error localization are partially decoupled metrics, and reporting only the first inflates apparent PRM quality.

PRMBench and broader evaluation

Song et al. 2025 ("PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models") extended the evaluation effort with a broader set of probe tasks covering process-level reward model behavior across multiple dimensions — simplicity preservation, soundness, sensitivity to step ordering, robustness to surface-level perturbations. The general finding is that PRMs developed primarily through best-of-N optimization on MATH-style benchmarks have brittle behavior on probes that explicitly test step-level reasoning quality.

The Qwen "Lessons" critique

Zhang et al. 2025 ("The Lessons of Developing Process Reward Models in Mathematical Reasoning"), from the Qwen team, made the methodological critique most pointedly. The paper argues that:

Monte Carlo rollout-derived labels often generalize worse than human or LLM-judge labels, despite being cheaper, because the rollouts share blind spots with the generator.
Best-of-N final-answer evaluation can systematically inflate PRM quality when correct final answers arise from flawed processes. The PRM appears to be selecting good reasoning; it is actually selecting trajectories that happen to converge.
Robust PRM development requires evaluation pipelines that decouple final-answer correctness from process correctness, and that test PRMs on out-of-distribution reasoning patterns rather than only on the distribution they were trained on.

The cumulative effect of ProcessBench, PRMBench, and the Qwen lessons paper is that the headline numbers from earlier PRM work should be read with substantial caution. Many of those numbers are best-of-N final-answer accuracy on MATH-style benchmarks, and the field now has tools showing that this metric overstates the PRMs' actual stepwise reasoning ability.

Compute accounting

"PRMs improve over ORMs at fixed compute" is one of the most common claims in PRM advocacy and one of the most underspecified. Several quantities are typically not equal across compared systems:

Annotation budget. PRM800K cost substantially more in skilled annotation than outcome labels for a comparable number of solutions. Comparisons that fix "verifier training budget" but not "label collection budget" hide this cost.
Generator compute during data construction. Automated PRM pipelines rely on rollouts from intermediate states, which are themselves generator calls. A PRM trained on rollout-derived labels has consumed substantially more generator FLOPs than an ORM trained on outcome-only labels for the same number of problems.
Verifier query cost at inference. A PRM is typically called once per step; an ORM is called once per candidate. For long traces, this multiplies inference cost.
Candidate budget under best-of-N. Comparisons at fixed N hold candidate count constant. But the more candidates one generates, the more an ORM can recover what a PRM was supposed to provide by selection. The right comparison varies N for both reward models and asks where the Pareto frontier lies.
Total training cost in RL. PRM-as-RL-reward systems require both PRM training and policy training. Outcome-only RL systems only require policy training. Holding "policy training FLOPs" constant ignores PRM training cost.

The honest version of the compute claim is: under a particular budget specification, in a particular domain, with a particular generator, this PRM variant achieved X versus this ORM variant's Y. The general claim that PRMs win at fixed compute does not have a clean public demonstration that survives careful accounting across all of these axes simultaneously.

Reasoning-model RL training and the active disagreement

The largest counterweight to "PRMs are the next step in reasoning supervision" is the public technical reports from 2024–2025 reasoning models. These are the most consequential evidence about what frontier labs actually use, and they cut against necessity claims.

DeepSeek-R1

DeepSeek-AI 2025 ("DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning") reports a reasoning model trained with large-scale RL using rule-based rewards — accuracy reward (the final answer either checks against a known solution or it does not) and format reward (the model followed the required output structure). The paper is explicit that DeepSeek-R1-Zero, the pure-RL variant, did not use neural outcome or process reward models for reasoning. The stated reasons are concrete:

Neural reward models are susceptible to reward hacking. A policy trained against a fixed neural reward model can find adversarial inputs that score well without satisfying the underlying objective.
Retraining the reward model in the loop adds substantial complexity to the training pipeline.
For verifiable tasks, the rule-based reward is unambiguous in a way that a learned reward model cannot match.

R1-Zero achieved strong reasoning performance on math and code benchmarks from pure rule-reward RL on a base model, with no SFT bootstrap and no PRM. The full DeepSeek-R1 added a small SFT stage and additional RL stages but retained the rule-reward approach for reasoning tasks.

This is the cleanest public counterexample to "frontier reasoning requires PRMs." It does not show that PRMs are useless; it shows that for the domains DeepSeek targeted, rule-based outcome rewards plus scaled RL plus long chains of thought were sufficient to produce a competitive reasoning model.

Kimi k1.5

Kimi Team 2025 ("Kimi k1.5: Scaling Reinforcement Learning with LLMs") is the second major report in this lineage. The paper presents an RL-based reasoning system and explicitly notes that it does not rely on Monte Carlo tree search, value functions, or process reward models — a deliberate simplification claim. The k1.5 architecture instead leans on long-context RL with a carefully designed policy optimization procedure and an emphasis on reasoning-trace length as a controllable resource.

Like DeepSeek-R1, k1.5 supports the narrower claim: outcome-style supervision plus scaled RL plus careful policy optimization can produce frontier-competitive reasoning without explicit PRMs.

OpenAI o-series

OpenAI's o1 and o3 technical disclosures provide weaker evidence in either direction because they do not document the training pipeline in the detail that DeepSeek and Kimi did. Public statements describe "reasoning RL" and "thinking time" as key components but do not commit publicly to either PRM-centric or outcome-only training. The most defensible reading of the o-series public record is: it is consistent with either PRM use or outcome-only RL, and the field cannot adjudicate from outside.

What the counterexamples do and do not show

The conservative reading of DeepSeek-R1 and Kimi k1.5 is:

They refute the necessity claim. "Frontier reasoning requires PRMs" is falsified by the existence of frontier-competitive reasoning models trained without explicit PRMs.
They do not refute the usefulness claim. Both papers operate in domains where rule-based outcome verification is reliable (math, code with unit tests). They do not show that PRM-style dense feedback would not help in less verifiable domains.
They redirect the burden of evidence. The pro-PRM camp now needs to argue not "PRMs help" but "PRMs help under matched compute and matched access to outcome verifiability" — a substantially harder claim.

The aggressive reading — that PRMs are now obsolete — is not supported. Both systems benefit heavily from the fact that final-answer verification is cheap and reliable in their target domains. For tasks where final outcomes are delayed, ambiguous, multi-faceted, or where wrong-for-the-right-reasons answers are unacceptable, the PRM case remains intact in principle even though the public empirical record is thin.

The PRM/ORM design space, summarized

The following table organizes the design choices and their trade-offs. It is not a ranking; it is a coordinate system.

Reward type	Label source	Granularity	Search use	RL use	Hacking risk	Domain transfer	Strength of public evidence
Cobbe-style ORM	Final-answer check	Trajectory	Best-of-N reranker	Sparse RL reward	Low (binary, hard to hack)	High in verifiable domains	Strong (GSM8K, MATH best-of-N)
Lightman PRM	Human step labels	Step	Best-of-N reranker, tree search	Dense RL reward (untested in Lightman)	Medium (step verifier can be exploited)	Limited beyond MATH-style problems	Strong on MATH best-of-N, narrow
Math-Shepherd PRM	Rollout-derived	Step	Reranker, search guide	Dense RL reward	Medium-high (shares blind spots with generator)	Limited beyond math	Moderate; growing
Process advantage verifier (PAV)	Rollout-derived progress estimate	Step (advantage)	Strong for search and RL	Strong	Medium (progress is a less hackable target than correctness)	Unclear; mostly math-tested	Moderate; recent
Implicit PRM	Derived from outcome objective	Token/step (implicit)	Limited	Native to certain RL methods	Inherits outcome hacking risk	Inherits outcome scaffolding	Moderate; method-dependent
Rule-based RLVR	Deterministic rule on final answer	Trajectory	Not used	Strong (DeepSeek-R1, Kimi k1.5)	Very low	Limited to rule-verifiable domains	Strong; recent

Several patterns are visible from this layout. Outcome-style rewards (Cobbe ORM, rule-based RLVR) are robust to hacking and easy to scale, but they only work where outcomes are verifiable and where reasoning quality conditional on correct outcome is not separately important. PRM variants offer denser feedback and better step-level discrimination, at the cost of more expensive label construction and higher hacking risk. PAVs occupy a useful middle position — they inherit the outcome-derived label source's scalability while learning a target (progress) that is less directly exploitable than local correctness.

Failure modes

The risks of PRM-centric training and inference are not abstract; they have concrete failure shapes that have appeared in the literature.

Reward hacking against the step verifier. A policy optimized against a fixed PRM learns features the PRM rewards. If the PRM rewards canonical solution style — well-formatted steps, standard algebraic manipulations, conventional phrasings — the policy learns to produce traces that match this style regardless of correctness. The visible symptom: PRM score rises during RL training while held-out accuracy stagnates or declines.

Reasoning theater. A subtler version of reward hacking. The policy learns to produce traces that look like careful reasoning to the PRM — appropriately segmented, with plausible intermediate steps — but the steps are post-hoc rationalizations rather than the actual computation that produced the answer. The trace becomes a performance of reasoning rather than a record of it. This is harder to detect because individual steps pass step-level checks; only their causal relationship to the answer is broken.

Calibration drift. PRMs trained on one distribution of solutions can have miscalibrated confidence on out-of-distribution solutions. A PRM trained on MATH-style proofs may score a valid but unusually-structured solution as low-quality, or score a confidently wrong solution as high-quality if it matches surface features of correct training solutions.

Search-induced selection bias. When a PRM is used to guide search, the resulting candidate distribution differs from the unconditioned generator distribution. If the PRM is then retrained on these PRM-selected candidates, it can drift toward rewarding whatever the PRM already preferred, amplifying its own biases.

First-error sensitivity. Step labels for steps after a first error are often undefined — the trace is on a wrong premise, so "correctness" of subsequent steps is ambiguous. Different labeling conventions for this case produce systematically different PRMs and complicate cross-paper comparisons.

Chain-of-thought access mismatch. Frontier reasoning systems often hide raw chains of thought from users for safety or product reasons. A PRM trained during model development on visible chains of thought does not automatically translate to deployed inspectability — process supervision during training is a different artifact from process transparency at deployment.

Generator-verifier coupling. When automated PRMs are constructed from rollouts of the same generator they will be used to score, the verifier inherits the generator's blind spots. Errors the generator reliably makes are errors the rollout-derived labels reliably mislabel.

These failure modes are not arguments against PRMs. They are arguments that "use a PRM" is not a free improvement — it imports a new optimization target with its own pathologies, which then need to be evaluated and mitigated. The literature's slow accumulation of these failure modes is part of why the field has not collapsed onto PRM-centric training.

When PRMs help, when they don't

Synthesizing the evidence, the conditional thesis takes a more specific shape. PRMs are likely to provide net benefit when:

Intermediate states are meaningful and labelable. Math proofs, code with stepwise tests, formal theorem proving, multi-step tool use with checkpoints. Reasoning where a "step" is a stable enough object that local correctness can be assessed.
First-error localization matters. Settings where finding the earliest mistake is more valuable than ranking complete trajectories, e.g., automated tutoring, code review, formal verification.
Final outcomes are expensive to verify. Search problems where a final verifier is costly (theorem proving with proof checkers, long-horizon agent tasks with delayed reward) and where pruning bad prefixes early saves substantial compute.
Reasoning fidelity matters independent of final accuracy. Oversight, alignment monitoring, scientific applications where wrong-for-the-right-reasons answers are unacceptable.
Search dominates inference compute. Tree search and beam search benefit from value estimates over partial trajectories in a way that best-of-N selection does not.

PRMs are likely to be redundant or actively unhelpful when:

Outcomes are cheaply and reliably verifiable. Math problems with known answers, code with executable tests, games with score functions. Rule-based RLVR captures most of the value without the labeling overhead.
The training signal needs to scale to many domains. Process supervision is domain-bound; constructing a new PRM for each new task domain is expensive relative to outcome-only training.
Reward hacking is the binding constraint. When the optimization budget is large enough that hacking against a neural reward model is a real risk, simpler outcome signals are more robust.
Generator quality is the bottleneck. If the generator's candidate set rarely contains a correct trajectory, no reward model can help. Compute spent on PRM training would yield more on generator improvement.
Inference budget is dominated by generation, not selection. When candidates are expensive and there are few of them, the marginal value of better selection is small relative to better generation.

The frontier-reasoning question — does a model competitive with o1/o3/R1 require process supervision? — sits in an awkward position relative to these criteria. The successful public examples (DeepSeek-R1, Kimi k1.5) work in domains where outcomes are cheaply verifiable and where rule-based RLVR is therefore a strong baseline. The cases where PRMs are most theoretically attractive — open-ended scientific reasoning, long-horizon agentic tasks, oversight-sensitive applications — are also the cases where public empirical evidence is thinnest in either direction.

The decisive experiment that does not yet exist

A clean adjudication of the necessity question would require an experiment with these properties:

Matched generator. The same base model (publicly available, large enough to be reasoning-capable) used across all conditions.
Matched total compute. A fixed budget covering: label/data construction (including all rollouts), reward-model training, policy training, and inference at evaluation. Reported as token counts and FLOPs, not as "training runs."
Multiple conditions. ORM best-of-N; human PRM (or distilled human PRM) best-of-N; Math-Shepherd-style automated PRM; process advantage verifier; implicit PRM derived from outcome training; pure rule-based RLVR with scaled RL and search.
Multiple domains. Math (MATH500, AIME-style held-out); code (LiveCodeBench, SWE-bench-style real-world tasks); proof (miniF2F, ProofNet); some less-verifiable domain (scientific reasoning, structured planning).
Multiple metrics. Final-answer accuracy; first-error localization (ProcessBench-style); trace validity conditional on correct final answer; calibration; reward-hacking incidence during RL; performance scaling with inference compute.

A result that would meaningfully move the needle: under matched total compute, PRM/PAV systems show a 10%+ relative gain in final accuracy or a 2x compute efficiency improvement over ORM/RLVR baselines, with equal or better first-error localization and no increase in invalid-reasoning correct answers, across at least three domains.

This experiment has not been run publicly. Component results exist — Lightman et al. on MATH best-of-N, Math-Shepherd on math RL, Setlur et al. on PAV efficiency, ProcessBench on first-error localization, DeepSeek-R1 on rule-based RLVR — but no single study spans the design space, the metrics, and the compute accounting. Until something resembling this experiment appears in the public record, the necessity question is not resolvable from outside frontier labs, and the strongest defensible position is the conditional thesis.

What the public reports actually adjudicate

It is worth being precise about which questions the existing evidence does and does not settle.

Settled by Cobbe et al. 2021: Verifier-guided selection improves over single-sample generation in math word problems. This is robust and reproduced.

Settled by Uesato et al. 2022: Process and outcome supervision can produce similar final-answer accuracy on GSM8K, but they differ in the rate of trace errors. The metric you choose determines which looks better.

Settled by Lightman et al. 2023: A human-process-supervised reward model can substantially outperform an outcome-supervised reward model as a best-of-N selector on MATH, in a setup that does not match annotation budgets. This is the strongest single piece of PRM evidence.

Settled by Math-Shepherd and follow-ups: Automated step labels derived from rollouts can produce step verifiers that improve both verification and RL on math benchmarks. The labels are cheaper than human annotation; the resulting verifiers have their own pathologies.

Settled by ProcessBench / PRMBench / Qwen lessons: Best-of-N final-answer accuracy systematically overstates PRM stepwise reasoning quality. PRMs that look good on the former often look much worse on the latter.

Settled by Setlur et al. 2024: Progress-based step rewards (PAVs) can outperform correctness-based step rewards for both search and RL, suggesting that the right learned object is value/advantage, not local correctness.

Settled by DeepSeek-R1 and Kimi k1.5: PRMs are not necessary for frontier-competitive reasoning in domains with reliable rule-based outcome verification. Rule-based RLVR plus scaled RL is a sufficient alternative.

Not settled: Whether PRMs are necessary, useful, or counterproductive in domains where rule-based outcome verification is unreliable. Whether the gains reported in PRM papers survive matched total compute accounting. Whether PRMs reduce or amplify reward hacking under aggressive RL. Whether process supervision generalizes economically beyond math-style domains. What OpenAI's o-series actually uses internally.

The unresolved questions cluster around the same axis: the boundary between "useful technique in some domains" and "necessary technique at the frontier" is exactly where the public evidence is thinnest, and that gap is unlikely to close without more open frontier-scale technical reports.

The narrow synthesis

Process reward models are an important and well-evidenced technique for stepwise supervision of mathematical reasoning, with the strongest results in best-of-N verification and verifier-guided search. They have not been shown to be necessary for frontier reasoning, and recent reasoning-model reports demonstrate that outcome-based or rule-based reinforcement learning can produce frontier-competitive results in verifiable domains without explicit PRMs. The right way to read the field is as a design space over label source, granularity, and use mode, in which PRMs occupy a useful but bounded region — most defensible when intermediate correctness is meaningful, when first-error localization matters, when search dominates inference compute, and when reasoning fidelity is valued independently of final accuracy.

The temptation to overgeneralize from Lightman et al. into "process supervision is the future of reasoning" is the dominant interpretive error in the popular literature, and it should be resisted. The corresponding temptation to read DeepSeek-R1 as "PRMs are obsolete" is the dominant error in the opposite direction, and it should also be resisted. Both readings discard evidence the field has actually produced in favor of a tidier story than the data supports.

The honest position is conditional. The interesting open questions are about which conditions matter most for production reasoning systems — and whether the cleanest reasoning architectures will eventually treat the PRM/ORM choice as an implementation detail of search-and-RL rather than as a fundamental design commitment.

Companion entries

Core theory: - Outcome Reward Models and Verifier-Guided Selection - Reinforcement Learning from Verifiable Rewards (RLVR) - Credit Assignment in Multi-Step Reasoning - Reward Model Generalization and Out-of-Distribution Behavior

Practice: - Verifier-Guided Search at Inference Time - Best-of-N Reranking: Compute Trade-offs - Monte Carlo Tree Search for Language Model Reasoning - Automated Label Construction for Reasoning Datasets - Step Segmentation and Trace Annotation Conventions

Empirical evidence: - Lightman et al. 2023 and the PRM800K Dataset - Math-Shepherd and Rollout-Derived Step Labels - Process Advantage Verifiers (Setlur et al. 2024) - ProcessBench and First-Error Localization - PRMBench: Fine-Grained PRM Evaluation - Qwen's Lessons of Developing Process Reward Models

Reasoning model training: - DeepSeek-R1: Rule-Based Rewards for Reasoning RL - Kimi k1.5 and Simplified Reasoning RL - OpenAI o-Series Reasoning Architecture (Public Disclosures) - Long Chain-of-Thought and Test-Time Compute Scaling

Counterarguments and failure modes: - Reward Hacking in Neural Reward Models - Reasoning Theater: Faithfulness of Chain-of-Thought - Compute Accounting Fallacies in Reward Model Comparisons - Generator-Verifier Coupling and Circular Supervision - Domain Transfer Failures of Math-Trained Reward Models

Adjacent concepts: - Self-Consistency and Majority Voting in Reasoning - LLM-as-Judge for Reasoning Evaluation - Formal Verification and Proof-Assistant-Guided Generation - Agentic Reasoning and Long-Horizon Credit Assignment