Scalar and pairwise evaluation answer different questions: scalar judging asks “how good is this output against an anchored rubric?”, while pairwise judging asks “which of these two outputs is better?” The strongest evaluation systems treat them as complementary measurement channels rather than rivals: scalar scores expose absolute and dimension-level quality, while pairwise preferences expose relative choice behavior and ranking structure.
Coverage note: verified through May 19, 2026.
The core distinction
In LLM-as-a-Judge evaluation, the two dominant rating modalities are scalar pointwise ratings and pairwise comparative ratings. Scalar evaluation rates each output independently, usually on a Likert scale, a 0–5 or 0–10 scale, or a dimension-specific rubric. Pairwise evaluation presents two outputs for the same task and asks a judge to choose A, choose B, or sometimes declare a tie.
Both are legitimate measurement instruments. The error is treating either as “the” evaluation primitive. They differ in estimand, sampling cost, calibration burden, aggregation method, and failure mode.
| Question | Scalar / pointwise rating | Pairwise A-vs-B comparison |
|---|---|---|
| Primitive judgment | “How good is this output?” | “Which output is better?” |
| Typical output | Score, often per rubric dimension | Win/loss/tie or preference probability |
| Natural estimand | Absolute quality estimate | Relative preference / ranking |
| Cost for N outputs | O(N) per rubric dimension | O(N²) for complete pair graph |
| Strength | Absolute thresholds, dimension diagnostics, deployment gates | Discrimination between close outputs, reduced scale-calibration burden, ranking |
| Weakness | Scale drift, rater severity, ceiling effects, false interval assumptions | Sparse-graph instability, order bias, non-transitivity, aggregation-rule dependence |
| Natural aggregation | Mean, trimmed mean, mixed model, item-response model | Win rate, Elo, Bradley-Terry Model, Thurstone, Davidson/Rao-Kupper tie models |
| Best use | Safety, faithfulness, rubric coverage, go/no-go gates | Model triage, preference ranking, A/B tournamenting, leaderboard construction |
The LLM-evaluation literature itself has settled into this distinction. Zheng et al.’s MT-Bench / Chatbot Arena work explicitly contrasts pairwise comparison with single-answer grading: pairwise comparisons can reveal subtle relative differences but become less scalable as the number of models grows, while single-answer grading is more scalable but may miss fine-grained pair-specific differences and can fluctuate under absolute-score calibration changes. arXiv
What scalar evaluation measures
A scalar rating is a judgment of one output against a scale. In LLM practice, the scale is usually one of four forms:
-
Likert-style agreement or quality levels, such as 1–5.
-
Anchored numeric rubrics, such as 0–10 with explicit descriptions for 0, 3, 5, 7, and 10.
-
Dimension-level rubrics, such as factuality, instruction following, helpfulness, safety, style fit, epistemic hygiene, or boundary safety.
-
Composite scalar scores, often a mean or sum over rubric dimensions.
Scalar evaluation is attractive because it produces a value that can be interpreted as an absolute quality estimate. If a medical-answer evaluator rates factuality as 3/10, the system does not need to know whether another model was worse; the output is below threshold. This is the key distinction from pairwise preference. A pairwise result can tell us that A beats B, but not whether either output is acceptable.
The NLG evaluation literature has used rating and Likert scales heavily for years. Amidei, Piwek, and Willis reviewed 135 NLG evaluation papers and found rating/Likert scales widely used, while also documenting deviations from good measurement practice and calling for better scale design. ACL Anthology This matters for LLM evaluation because most LLM-as-judge scalar prompts inherit the same psychometric problems: unclear anchors, mixed dimensions, unreported rater agreement, and accidental treatment of ordinal categories as if they were clean interval measurements.
The right mental model is: scalar scores are not raw truth. They are scale-mediated measurements whose usefulness depends on anchor quality, judge calibration, rubric specificity, and score distribution.
What pairwise evaluation measures
A pairwise comparison asks the judge to compare two outputs under the same prompt. It may be forced-choice, or it may allow ties. It produces a local preference relation:
A≻B,B≻A,orA∼B.A \succ B,\quad B \succ A,\quad \text{or}\quad A \sim B.A≻B,B≻A,orA∼B. That local relation can then be aggregated into win rates, rankings, Elo-style ratings, or Bradley-Terry parameters. The classic statistical foundation is Bradley and Terry’s 1952 method of paired comparisons, which models the probability that one item beats another as a function of latent strengths. OUP Academic In modern LLM evaluation, Chatbot Arena popularized this pattern at scale: users compare two anonymous model responses, and the resulting human preferences are aggregated into model rankings. The ICML Chatbot Arena paper describes the platform as using pairwise comparison with crowdsourced human votes and reports over 240,000 votes in the analyzed data. Proceedings of Machine Learning Research
Pairwise evaluation is attractive because it matches the practical decision that many model developers face: “which candidate should we ship, fine-tune further, or discard?” It also avoids asking judges to map quality onto an absolute scale. A judge does not need to decide whether an answer is “7” or “8”; they only need to decide whether it is better than another answer.
That reduced calibration burden is the main empirical and methodological case for pairwise evaluation.
The empirical case for pairwise
Comparative judgments often discriminate better
The strongest human-evaluation argument for pairwise methods is not unique to LLMs. Human raters frequently struggle to maintain stable absolute-scale interpretations across items, time, and contexts. Comparative methods reduce this burden by making the reference item visible.
A useful predecessor is best-worst scaling, a comparative annotation method in which annotators choose the best and worst items from a small set. Kiritchenko and Mohammad directly compared best-worst scaling with rating scales for sentiment-intensity annotation and found that, with the same total number of annotations, best-worst scaling produced significantly more reliable results than rating scales. ACL Anthology Best-worst scaling is not identical to binary A-vs-B comparison, but it supports the broader methodological point: humans can often make comparative distinctions more reliably than calibrated absolute ratings.
The LLM-as-judge literature shows a parallel pattern, though not uniformly. Zheng et al. found that strong LLM judges, after addressing biases, could match human evaluations at over 80% agreement in their MT-Bench / Chatbot Arena setup, with pairwise and single-answer grading both studied as evaluation modes. arXiv Liu et al.’s PAIRS work argues more directly for pairwise preference in LLM evaluators: it reports that score-based LLM evaluators exhibit distributional misalignment with human ratings and that pairwise-preference search can improve alignment with human judgment in long-form generation evaluation. OpenReview
The clean claim is not “pairwise is always better.” It is: when the task is to discriminate among plausible alternatives, comparative judgment often provides more usable signal than absolute scoring.
Pairwise sidesteps absolute-scale calibration
Scalar scores require judges to interpret the scale consistently. If one judge treats 7/10 as “solid” and another treats 7/10 as “barely acceptable,” scalar means blur rater severity with output quality. LLM judges add another complication: their numeric-score priors may differ from human-score priors. Liu et al. frame this as a misalignment between LLM and human evaluation standards under Likert-style scoring, and show that calibration does not necessarily close the gap. OpenReview
Pairwise comparison reduces that burden. The judge need not place outputs on a global scale; it only chooses between two visible alternatives. This does not eliminate bias, but it changes the bias surface. A judge can still prefer verbosity, flattery, confident formatting, or position A. Yet the problem is no longer “what does 8 mean?” It becomes “what features drive the preference relation?”
This is a major practical advantage in fast model-development loops. When comparing prompt variants, decoding settings, retrieval policies, or fine-tune checkpoints, the development question is often local: which candidate is preferred under a defined slice? Pairwise judging maps directly to that question.
Pairwise data maps naturally to ranking systems
Pairwise outcomes produce graph data: nodes are models or output variants; edges are preferences. That graph can be aggregated using simple win rates, but richer models are usually better. The two best-known families are Elo Ratings for LLMs and Bradley-Terry Model aggregation.
Elo-style ratings estimate relative strength from match outcomes. Chatbot Arena initially used Elo-like methods because they are intuitive and familiar from chess-style ranking. The Arena team later described moving toward Bradley-Terry maximum-likelihood estimation for more stable batch inference and better confidence intervals, while noting that LLM ranking differs from chess because the full history of model matchups can often be processed centrally and many model versions are intended to be static. LMSYS Org
The modern Arena statistical framing makes the connection explicit: Chatbot Arena uses Bradley-Terry-style inference for model strength, and extended Arena score formulations express pairwise outcomes as logistic-regression-style estimates whose coefficients are often transformed onto an Elo-like display scale. Arena
The essential Bradley-Terry form is:
P(i≻j)=σ(θi−θj),P(i \succ j)=\sigma(\theta_i-\theta_j),P(i≻j)=σ(θi−θj), where θi\theta_iθi and θj\theta_jθj are latent strengths and σ\sigmaσ is the logistic function. This makes pairwise judgments ideal for rank estimation, provided the graph is sufficiently connected and the sampling process is defensible.
The empirical case for scalar
Scalar scores provide absolute quality estimates
Pairwise preference cannot answer “is this safe enough?” or “does this meet the factuality threshold?” without an external anchor. If A beats B 70% of the time, A may be excellent, mediocre, or merely less bad. Scalar evaluation can express that both A and B are unacceptable, or that both are acceptable but one is marginally better.
This matters most for Deployment Go-No-Go Evaluation. Deployment gates usually require minimum absolute standards: factuality above a threshold, refusal behavior within policy, hallucination risk below a threshold, latency or cost within budget, and style constraints satisfied. Pairwise comparison can support those gates, but it cannot replace them unless it includes anchored baselines such as “candidate versus known acceptable reference” or “candidate versus minimum viable output.”
Scalar scoring is also better for dimension diagnosis. A pairwise judge may choose B over A, but unless the prompt forces structured rationale, the win may reflect any mixture of factuality, concision, completeness, tone, confidence, formatting, or hidden judge preference. A scalar rubric can expose that A was more factual, B was more helpful, and both were weak on uncertainty calibration.
Scalar evaluation is O(N), not O(N²)
For N outputs, independent scalar scoring requires O(N) judgments per dimension or per rubric pass. Full pairwise comparison requires:
N(N−1)2\frac{N(N-1)}{2}2N(N−1) comparisons, or twice that if AB/BA counterbalancing is used. Zheng et al. make this scalability issue explicit: pairwise comparison may lack scalability as the number of players grows because possible pairs grow quadratically. arXiv Liu et al. make the same computational point in the ranking setting, noting that complete non-transitive rank aggregation requires O(N²) pairwise comparisons and exact optimal aggregation is NP-hard under the general formulation. OpenReview
There are workarounds: Swiss tournaments, active sampling, uncertainty-guided pair selection, approximate sorting, and Bradley-Terry graph designs. But those workarounds change the estimand. They no longer observe the full pair graph; they infer a ranking from a sparse graph under model assumptions.
Scalar evaluation is therefore more efficient when the goal is to compare many outputs against the same rubric. If a team needs to evaluate 500 outputs on factuality, policy compliance, and answer completeness, scalar scoring is often the only practical first pass.
Scalar can be more robust against some pairwise-specific distractors
The empirical case for pairwise has a counterweight: pairwise comparison can amplify spurious relative preferences. Recent work comparing absolute scores and pairwise preferences finds that pairwise LLM-evaluation protocols can be more vulnerable to “distracted evaluation,” where generator outputs exploit features favored by the judge. In that study, pairwise preferences flipped in about 35% of cases under distractors, compared with about 9% for absolute scores. OpenReview Another BlackboxNLP 2025 paper found pairwise evaluators performed better on MT-Bench but worse than pointwise evaluators on adversarial LLMBar samples, arguing that pairwise comparison can amplify biased preferences under adversarial conditions. ACL Anthology
This is the strongest reason not to treat pairwise as a universal upgrade. Pairwise evaluation reduces one calibration problem but can intensify contrastive salience problems. If the judge is drawn to fluency, length, confidence, or rhetorical polish, putting two outputs side by side may make the distractor more salient, not less.
Methodological tensions
Pairwise tension: incomplete graphs can be unstable
Pairwise rankings are only as reliable as the comparison graph. A complete graph is expensive; an incomplete graph is model-dependent. If the graph under-samples certain matchups, over-samples popular models, or contains influential edges near the top of the leaderboard, small data changes can move rankings.
Recent robustness work on LLM leaderboards makes this concrete. Huang et al. propose testing Bradley-Terry-style LLM rankings by dropping a worst-case small fraction of preference data and report that dropping just 0.003% of human preferences can change the top-ranked model on Chatbot Arena. arXiv A May 2026 perturbation framework similarly reports that modern pairwise-comparison leaderboards can be non-robust under sub-1% targeted perturbations, changing the top-ranked model, degrading Kendall’s tau, or altering confidence intervals. arXiv
The lesson is not that pairwise leaderboards are useless. It is that sparse pairwise graphs need robustness audits. Reported rank should be paired with uncertainty, graph coverage, influential-edge analysis, and sensitivity to drop/add/flip perturbations.
Pairwise tension: social-choice pathologies are real
Pairwise preferences do not necessarily compose into a stable global order. LLM judges may prefer A over B, B over C, and C over A. Humans can do the same. Liu et al. explicitly note that LLM pairwise preferences do not guarantee transitivity and that ranking from non-transitive pairwise probabilities becomes a rank-aggregation problem. OpenReview
This connects LLM evaluation to Condorcet Paradox, Borda Count, and broader social-choice theory. Strictly speaking, the “Borda paradox” is not a property of pairwise comparison alone: Borda is a positional aggregation rule. The relevant point is broader: different aggregation rules can select different winners from the same underlying preferences, and pairwise-majority winners, Borda winners, and Kemeny-style rankings need not agree. Saari’s social-choice analysis covers cycles, Borda/Condorcet conflict, and aggregation paradoxes in three-alternative settings. math.uci.edu Other voting-theory work notes the classic discrepancy where a Borda count may fail to elect a Condorcet winner. ace-economics.fi
For LLM output judging, this means “pairwise win rate” is not a magic preference oracle. It is a social-choice dataset plus an aggregation rule. The aggregation rule should be declared.
Pairwise tension: position, verbosity, and contrast effects
Pairwise evaluation is vulnerable to presentation artifacts. Zheng et al. report strong position bias among LLM judges and show that swapping answer order can flip judgments; their analysis also discusses verbosity bias, where longer outputs are favored even when not better. arXiv The practical mitigation is AB/BA counterbalancing: judge both A/B and B/A, then require consistency or average over order.
But AB/BA doubles cost. It also does not solve all pairwise artifacts. If the judge prefers length, formality, disclaimers, or confident enumeration, the side-by-side format may intensify the preference. Pairwise prompts should therefore include explicit anti-verbosity instructions, length controls where possible, and tie options for cases where neither output is materially better.
Scalar tension: ceiling effects and range restriction
Scalar evaluation fails when judges cluster scores near the top of the scale. If nearly every output receives 4.4–4.8 on a 0–5 rubric, scalar means may be too compressed to distinguish variants. This is not theoretical. PsycheEval v0.1 reported scalar means clustered between 4.0 and 4.8 on a 0–5 scale, making many cell-to-cell differences descriptive rather than inferential; v0.2 moved to an anchored 0–10 rubric specifically to recover headroom. Ashita Orbis Liu et al. also note an evaluation setting where human scores were heavily concentrated at the top, with 86.7% of summaries receiving a score of 5, making pairwise ranking difficult because the scalar gold signal itself had little spread. OpenReview
Ceiling effects are especially common when outputs are generated by frontier models on non-adversarial prompts. If all outputs are fluent, safe, and superficially helpful, generic scalar rubrics saturate. Anchors must be sharpened, prompts must be harder, or the evaluation must shift to pairwise preference, targeted stress cases, or error-seeking rubrics.
Scalar tension: ordinal scales are often treated as interval scales
A Likert score is usually ordinal: 5 is better than 4, but the distance from 4 to 5 is not guaranteed to equal the distance from 2 to 3. In practice, evaluation teams average Likert scores anyway. That may be acceptable as an engineering approximation, but it should not be hidden. The more consequential the decision, the more important it is to justify aggregation: use anchored rubrics, inter-rater agreement, bootstrap intervals, item-response models, or hierarchical models rather than naked means.
Amidei et al.’s review is relevant here because it is not anti-rating-scale; it is anti-sloppy-rating-scale. Rating scales are useful, but only when scale design and analysis match the claims being made. ACL Anthology
Aggregating pairwise data: win rates, Elo, and Bradley-Terry
A pairwise dataset can be aggregated at several levels of sophistication.
Raw win rate
The simplest metric is:
win rate(i)=winsidecisive comparisonsi.\text{win rate}(i)=\frac{\text{wins}_i}{\text{decisive comparisons}_i}.win rate(i)=decisive comparisonsiwinsi. This is easy to understand but fragile. It ignores opponent strength. A model that beats weak opponents may look better than a model with a lower win rate against stronger opponents. It also handles graph imbalance poorly.
Raw win rate is most defensible in controlled head-to-head experiments where the pair set is fixed in advance: model A versus model B over the same prompt distribution, same judges, same tie policy, and counterbalanced order.
Elo-style ratings
Elo updates a rating after each match based on expected versus observed outcome. Its advantage is interpretability and online updating. Its disadvantage is path dependence: ratings can depend on match order, K-factor choices, and changing model populations. Arena’s own methodology notes that classic Elo assumes dynamically changing player strength, whereas many LLM model versions are treated as static and can be estimated from the full comparison history. LMSYS Org
Elo is useful for live systems, but batch Bradley-Terry is usually cleaner for post-hoc statistical inference.
Bradley-Terry models
The Bradley-Terry model estimates latent strengths from pairwise outcomes. It is the natural statistical object behind many LLM preference leaderboards. Arena describes its move to Bradley-Terry maximum-likelihood estimation as a way to improve ranking quality and confidence estimates; it also notes a simple tie treatment in which ties count as half a win and half a loss. LMSYS Org
Bradley-Terry is not assumption-free. It assumes a logistic relationship between latent strength differences and pairwise win probability. It also assumes that the relevant comparison context is captured by the model specification. If prompt category, judge family, answer length, or user population changes the outcome distribution, a single global strength parameter may hide important slice-level structure.
Tie-aware extensions
Ties should not be treated as an afterthought. A forced-choice setup can inflate meaningless preferences when two outputs are effectively indistinguishable. A tie option can reduce noise but introduces its own calibration problem: some judges are tie-happy, others almost never tie.
There are three common approaches:
| Tie policy | Meaning | Risk |
|---|---|---|
| No ties allowed | Every comparison yields A or B | Artificial preferences, especially among near-identical outputs |
| Tie as half win / half loss | Simple Arena-style pragmatic encoding | Treats all ties as equally informative |
| Explicit tie model | Davidson, Rao-Kupper, or similar extensions | More parameters, more data required |
For LLM evaluation, tie handling should be chosen before data collection. Post-hoc tie manipulation is leaderboard manipulation.
Relationship to human-rating practice
The scalar-versus-pairwise split mirrors a long-standing distinction in human-rating methodology: Likert/rating scales versus forced-choice/comparative paradigms.
Likert-style methods are efficient and interpretable, but depend on stable scale use. They are natural when the evaluation question has an absolute standard: satisfaction, agreement, severity, quality, or risk. Comparative methods are natural when raters struggle with absolute calibration or when relative preference is the target. Best-worst scaling, paired comparison, ranking tasks, and forced-choice psychometrics all live in this family.
The LLM setting adds two twists.
First, LLM judges are both raters and language models. They can explain judgments, but their explanations may rationalize hidden priors. A scalar LLM judge may overuse mid-to-high scores; a pairwise LLM judge may prefer the first answer, the longer answer, or the answer that sounds more like its own training distribution.
Second, LLM outputs are high-dimensional artifacts. “Better” may mean more factual, more concise, more emotionally attuned, more policy compliant, more creative, less sycophantic, or better matched to a user profile. Scalar rubrics can decompose these dimensions. Pairwise preferences collapse them into a choice unless the comparison prompt explicitly asks for dimension-specific pairwise judgments.
This is why high-quality evaluation designs increasingly combine both.
Design pattern: use both channels
The strongest practical design is not “scalar or pairwise.” It is scalar plus pairwise, with disagreement treated as signal.
A useful pattern is:
-
Generate candidate outputs under controlled conditions.
-
Score each output independently on anchored scalar rubric dimensions.
-
Run pairwise comparisons on the same outputs, preferably same prompt distribution and judge pool.
-
Report scalar means and pairwise preference rates separately.
-
Reconcile the channels by asking where they agree, where they diverge, and what each divergence implies.
PsycheEval is a concrete example of this pattern. PsycheEval v0.1 used three measurement channels: a 0–5 Likert scalar rubric over ten dimensions, a red-flag taxonomy, and a forced-choice A/B/tie pairwise channel. The author explicitly notes that the pairwise channel was same-author-only, isolating condition effects rather than author effects, and that this scope qualifier limits what the pairwise results can answer. Ashita Orbis
PsycheEval v0.2 then hardened the design: it added an anchored 0–10 rubric to break the v0.1 ceiling, length-control conditions, a harder scenario set, and counterbalanced AB/BA judging. The run produced 1,680 assistant outputs, 3,949 anchored scalar judgments, and 3,243 raw pairwise records. Ashita Orbis The important methodological result was not merely a winner; it was the discovery that an initial pairwise headline did not survive scalar-pairwise reconciliation and AB/BA position-bias audit. Ashita Orbis
That is the right attitude. Channel disagreement is not an embarrassment. It is one of the main reasons to run both channels.
Interpreting scalar-pairwise disagreement
| Scalar result | Pairwise result | Likely interpretation | Audit |
|---|---|---|---|
| Scalar and pairwise agree | Same winner or same null | Stronger evidence, though still slice-bound | Check CIs, judge agreement, and length |
| Scalar flat, pairwise prefers B | Preference may be style, salience, length, or unmodeled rubric dimension | AB/BA, length control, rationale coding | |
| Scalar favors A, pairwise favors B | Rubric and preference target conflict | Inspect dimensions; decide which target matters | |
| Scalar favors A, pairwise ties | Difference may be real but not preference-salient | Check effect size and threshold relevance | |
| Scalar ceiling, pairwise separates | Rubric lacks headroom | Add harder prompts or sharper anchors | |
| Pairwise unstable, scalar stable | Sparse graph, order bias, or near-tie outputs | More edges, tie model, cluster bootstrap |
The key operational principle is: do not collapse disagreement too early. If scalar says one output is safer but pairwise says another is preferred, that is not a rounding error. It may reveal that users prefer unsafe confidence, that the rubric overweights a dimension users do not value, or that the judge is being seduced by rhetoric.
Methodological extensions
Anchored scalar rubrics
A scalar rubric should not merely say “rate helpfulness from 1 to 10.” It should anchor levels with behavioral descriptions. For example:
| Score | Anchor |
|---|---|
| 0 | Dangerous, false, or directly violates task constraints |
| 3 | Partially relevant but misses core requirements or contains serious uncertainty failures |
| 5 | Adequate but generic; satisfies basic request with notable omissions |
| 7 | Strong, mostly complete, minor issues |
| 10 | Excellent under the rubric; precise, complete, calibrated, and task-fit |
Anchors reduce but do not eliminate rater drift. For LLM judges, anchors also make the prompt more legible: the model has concrete criteria to map onto.
Good scalar rubrics also separate dimensions. “Overall quality” is useful as a summary, but development needs decomposed signal: instruction following, factuality, reasoning validity, calibration, harmlessness, user-fit, and presentation.
Tied-pair handling
Pairwise prompts should usually allow ties unless the target application truly requires forced choice. Ties are especially important when evaluating frontier-model outputs on easy prompts, where many pairs differ only cosmetically.
But ties must be modeled. Report:
-
tie rate by judge,
-
tie rate by condition pair,
-
tie rate by prompt category,
-
whether ties were excluded, counted as half wins, or modeled explicitly,
-
whether “no material difference” was distinguished from “both bad.”
A tie between two excellent outputs and a tie between two unacceptable outputs are not the same evaluation fact. This is another reason scalar and pairwise should be paired: scalar can identify the quality level of tied outputs.
AB/BA counterbalancing
For pairwise evaluation, AB/BA counterbalancing should be default when cost allows. Zheng et al.’s position-bias findings are strong enough that uncounterbalanced pairwise results should be treated as preliminary unless the model/judge setup has been independently validated. arXiv
The minimal pattern is:
-
Present A left / B right.
-
Present B left / A right.
-
Treat consistent choices as decisive.
-
Treat flips as ties, uncertain cases, or evidence of position sensitivity.
-
Report the flip rate.
PsycheEval v0.2 is an example of why this matters: a planned pairwise headline did not survive counterbalanced AB/BA judging. Ashita Orbis
Cluster-bootstrap confidence intervals
Both scalar and pairwise evaluation often violate independence assumptions. The same prompt, persona, scenario, author model, or judge may appear in many records. Treating every judgment as independent inflates effective sample size.
A better default is a cluster bootstrap over the natural repeated-measure unit. In LLM-output judging, likely clusters include:
-
prompt,
-
user/persona,
-
scenario family,
-
author model,
-
judge model,
-
persona × scenario × author cell,
-
benchmark item.
PsycheEval v0.1 explicitly warns that Wilson intervals on pairwise records assume independence even though the same persona × scenario × author cell appears in multiple condition rows; it suggests cluster bootstrap intervals as a correction. Ashita Orbis
The same logic applies to scalar means. A scalar table without uncertainty intervals is descriptive. That may be enough for triage, but it is not enough for strong claims.
Slice-level reporting
Global means hide failure modes. Report scalar and pairwise results by:
-
prompt family,
-
difficulty,
-
output length bucket,
-
judge model,
-
author model,
-
user segment,
-
safety category,
-
language,
-
retrieval/no-retrieval condition.
A model that wins globally may lose on high-stakes slices. A model that scores high on helpfulness may fail calibration. Pairwise and scalar both become more useful when sliced.
Choosing by evaluation stakes
The right modality depends on the decision.
| Evaluation context | Preferred primary channel | Why | Secondary channel |
|---|---|---|---|
| Prompt or checkpoint triage | Pairwise | Fast relative choice among variants | Scalar dimensions for diagnosis |
| Regression testing | Scalar thresholds plus targeted pairwise | Need absolute pass/fail and relative regression signal | Pairwise against previous release |
| Public leaderboard | Pairwise with Bradley-Terry / Elo-style aggregation | Ranking is the object | Scalar task diagnostics |
| Safety evaluation | Anchored scalar rubric | Need absolute risk threshold | Pairwise only for comparing mitigations |
| Product UX preference | Pairwise or best-worst | User preference is comparative | Scalar satisfaction/risk ratings |
| Deployment go/no-go | Scalar gates | “Better than baseline” is not enough | Pairwise against baseline and reference outputs |
| Research claim about model quality | Both | Each has distinct validity threats | Channel reconciliation required |
For model-development triage, pairwise is often the right first instrument. It answers the engineering question quickly: which candidate is preferred on this slice? It is especially good for local comparisons among prompt variants, retrieval policies, or fine-tuned checkpoints.
For deployment go/no-go, scalar should usually be primary. A deployment decision needs absolute constraints: safety, factuality, legal compliance, calibration, refusal correctness, and task completion. Pairwise can show that candidate B is better than candidate A, but a better unsafe system is still unsafe.
For high-stakes evaluation, the preferred design is both channels plus human calibration. LLM judges may be useful for scale, but human panels, adversarial cases, and audit trails are needed when the output domain has real-world risk.
Practical default recipe
A robust evaluation harness should define the estimand before choosing the modality.
If the estimand is absolute quality, use anchored scalar scoring. Add pairwise comparisons only where relative preference matters.
If the estimand is relative preference, use pairwise comparisons. Add scalar scoring to determine whether the preferred output is actually acceptable and why it won.
If the estimand is rank among many systems, use pairwise graph sampling plus Bradley-Terry or related aggregation. Report graph coverage, uncertainty, and robustness to perturbation.
If the estimand is rubric compliance, use scalar dimension ratings and error taxonomies. Pairwise comparison should not be the primary metric unless the rubric itself is comparative.
A defensible LLM-output judging report should include:
| Required field | Scalar | Pairwise |
|---|---|---|
| Prompt distribution | Yes | Yes |
| Judge identity/version | Yes | Yes |
| Rubric or comparison instruction | Yes | Yes |
| Scale anchors | Yes | Not applicable, unless dimension-pairwise |
| Tie policy | Not applicable | Yes |
| Order counterbalancing | Usually not | Yes |
| Uncertainty intervals | Yes | Yes |
| Cluster unit | Yes | Yes |
| Judge agreement | Yes | Yes |
| Slice-level results | Yes | Yes |
| Failure-mode audit | Yes | Yes |
The open question
The field has enough evidence to reject simple answers. Pairwise evaluation is often better for discrimination, preference ranking, and model-development triage. Scalar evaluation is often better for absolute quality, dimension diagnosis, and deployment thresholds. But the boundary is not fixed.
The unresolved question is not “which modality is best?” It is:
For a given task, judge type, output distribution, and decision stake, which modality has the lower decision error?
That is an empirical question. The answer may differ for summarization, code generation, tutoring, roleplay, medical QA, agent planning, and personalized assistance. It may differ for human judges and LLM judges. It may differ between normal and adversarial prompts. Recent work showing pairwise vulnerability to distractors is especially important because it complicates the now-common assumption that comparative judgment is always more reliable. OpenReview
A good evaluation system therefore treats scalar and pairwise as two partially independent sensors. Agreement increases confidence. Disagreement identifies the next audit.
References and primary sources
Bradley & Terry 1952 — method of paired comparisons. Foundational paired-comparison model for estimating latent item strength from pairwise outcomes. OUP Academic
Zheng et al. 2023 — MT-Bench, Chatbot Arena, and LLM-as-a-judge. Introduces MT-Bench and Chatbot Arena analysis, studies pairwise and single-answer grading, and documents position and verbosity biases in LLM judges. arXiv
Chiang et al. 2024 — Chatbot Arena ICML paper. Describes large-scale pairwise human-preference evaluation and ranking of LLMs via crowdsourced comparisons. Proceedings of Machine Learning Research
LMSYS / Arena methodology notes — Elo, Bradley-Terry, and tie handling. Explains the transition from online Elo-style updates to Bradley-Terry maximum-likelihood estimation and notes practical tie handling as half win / half loss. LMSYS Org
Arena extended Bradley-Terry / Elo notes. Describes extended Bradley-Terry and online Elo formulations for binary-comparison inference problems, including Elo-scale transformation of coefficients. Arena
Amidei, Piwek, and Willis 2019 — rating and Likert scales in NLG evaluation. Reviews 135 NLG papers using rating/Likert scales and recommends better scale practice. ACL Anthology
Kiritchenko and Mohammad 2017 — best-worst scaling versus rating scales. Directly compares comparative annotation with rating scales and finds best-worst scaling more reliable under equal annotation budgets. ACL Anthology
Liu et al. 2024 — PAIRS and pairwise preference in LLM evaluators. Argues that pairwise-preference search can reduce misalignment between LLM evaluators and human judgment relative to direct score-based evaluation, while also noting non-transitivity and O(N²) complete-comparison costs. OpenReview
Tripathi et al. 2025 — pairwise versus pointwise bias under distractors. Shows that pairwise protocols can be more vulnerable than absolute scoring to distractor features in LLM-based evaluation. OpenReview
BlackboxNLP 2025 — pairwise comparisons amplifying biased preferences. Finds pairwise evaluation performs better on MT-Bench but worse than pointwise evaluation on adversarial LLMBar samples. ACL Anthology
Leaderboard robustness work 2025–2026. Shows that small targeted perturbations to pairwise-preference data can change top LLM rankings under Bradley-Terry-style leaderboard aggregation. arXiv
PsycheEval v0.1 and v0.2. Reports a combined scalar, red-flag, and pairwise evaluation harness; v0.2 demonstrates the value of anchored scalar rubrics, scalar-pairwise reconciliation, cluster bootstrap thinking, and AB/BA position-bias audits. Ashita Orbis+3Ashita Orbis+3Ashita Orbis+3
Companion entries
Core theory: Bradley-Terry Model, Elo Ratings for LLMs, Thurstone’s Law of Comparative Judgment, Condorcet Paradox, Borda Count, Rank Aggregation, Item Response Theory
LLM evaluation practice: LLM-as-a-Judge, Chatbot Arena, MT-Bench, PsycheEval, Evaluation Harness Design, Model-Development Triage, Deployment Go-No-Go Evaluation
Scalar methods: Likert Scale, Anchored Rubrics, Rubric Dimension Design, Ceiling Effects, Rater Severity, Scalar Calibration
Pairwise methods: Pairwise Preference Learning, Forced-Choice Evaluation, Best-Worst Scaling, AB/BA Counterbalancing, Tie Handling in Pairwise Evaluation, Sparse Pairwise Graphs
Reliability and uncertainty: Cluster Bootstrap, Inter-Rater Agreement, Confidence Intervals for Evaluation, Leaderboard Stability, Perturbation Audits
Failure modes and counterarguments: Position Bias, Verbosity Bias, Distractor Features, Non-Transitive Preferences, Goodhart’s Law in Evaluation, Preference Versus Quality