Abstract. When a language model judges outputs produced by other language models, its score is conditional on who is judging whom. Halo effects are systematic shifts in score that track the relationship between author model and judge model — most directly when the judge scores its own outputs (exact-self), more diffusely when the judge and author share a provider or model family (same-provider), and minimally when the judge has no architectural affinity to the author (cross-provider). This article treats halo as a measurement-validity problem rather than a moralized bias problem: each cell estimates a different quantity, and the methodological question is which estimand a given benchmark claim is entitled to.
Coverage note: verified through May 19, 2026.
1. The taxonomy
Every LLM-as-judge record can be tagged with a relation between the model that produced the output (the author) and the model that scored it (the judge). Three relations matter for halo analysis:
| Relation | Definition | Example |
|---|---|---|
| Exact-self | Author model and judge model are the same exact model and version. | GPT-5.5 grades GPT-5.5. |
| Same-provider, non-self | Author and judge share a provider or model family but are different models. | Claude Opus 4.6 grades Claude Sonnet 4.6. |
| Cross-provider | Author and judge are from different providers / families. | Claude Sonnet 4.6 grades GPT-5.5. |
The taxonomy looks tidy and is operationally important, but the categories conceal three different causal mechanisms.
Exact-self preference can arise because the judge recognises its own generations (the most direct mechanism), because the judge and author share idiosyncratic latent criteria, because they interpret an ambiguous rubric the same way, or because of overlapping training data and post-training conventions. Same-provider preference can arise from shared post-training recipes, refusal policies, verbosity norms, and assistant-style fingerprints, but it can also arise from genuine quality differences between providers — an effect that has nothing to do with affinity. Cross-provider scores are simply what remains after the most obvious author–judge entanglements are removed; they are cleaner, not pure.
The taxonomy is therefore a partition of records, not a partition of causes. The methodological pay-off comes from analysing the cells separately, not from imagining that the cell label resolves the causal question.
A frequently-conflated fourth category — "all within-provider" — bundles exact-self with same-provider-non-self. The conservative scope drops the whole bundle; the more granular scope drops only exact-self and reports the remainder. Which is the right cut depends on what claim the benchmark is trying to make, not on a universal best practice. See LLM-as-Judge for the broader methodology that this article sits inside, and Construct Validity for the underlying epistemology.
2. Why the cells must be reported separately
The dominant methodological failure in LLM-as-judge evaluation is to publish a single aggregate judge score without exposing the author–judge cell structure. The number then estimates a mixture of three quantities at proportions that depend on which models happened to be in the panel.
It is more honest to fix the question first and pick the scope second. Three distinct estimands recur:
- What would this judge panel say, given exactly this set of authors and judges? This is the full-panel estimand. It is the right answer for product-internal questions ("does our deployed evaluator like our deployed model?"), and it is the wrong answer for general-quality claims, because the result is fully conditional on the panel composition.
- What would judges with no exact-self exposure say about these outputs? This is the exact-self-excluded estimand. It removes the most direct contamination pathway but retains same-provider-non-self records, which can carry shared-style preference without exact recognition.
- What would judges from a different provider say about these outputs? This is the cross-provider estimand. It is the conservative scope for public, cross-organisation claims because both exact-self and same-provider-non-self records are dropped. It is not unbiased; it changes the population, can reduce statistical power, and can penalise distinctive but legitimate provider styles.
These are not three approximations of one true number. They are three different numbers, defined over different sub-populations of records, answering different questions. A benchmark that reports only one of them has implicitly chosen the question without telling the reader. A benchmark that reports all three with sample sizes lets the reader see how much the answer depends on which judges were included.
The vocabulary that fits this framing is not "the biased number" versus "the corrected number." It is which estimand are we reporting, and what claim does it support? That is the spine of the rest of this article.
3. Empirical evidence
The empirical state of the literature is uneven across the three cells. Exact-self preference has the strongest direct evidence; same-provider preference is plausible and important but underidentified; cross-provider scoring is well-motivated as a robustness scope but not validated as a ground-truth oracle.
3.1 Exact-self preference
The most direct evidence is Panickssery, Bowman, and Feng 2024, LLM Evaluators Recognize and Favor Their Own Generations, which reports that LLM evaluators can recognise their own outputs at non-trivial rates and assign them higher scores than alternatives that human annotators rate similarly or better. The paper goes further than earlier "self-enhancement" framings by linking the preference to a measurable self-recognition capability rather than treating it only as a downstream rating artefact.
The earlier Zheng et al. 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena paper had already documented "self-enhancement bias" as one of several systematic LLM-judge failure modes alongside position bias, verbosity bias, and limited reasoning. MT-Bench is the wider methodological reference point for LLM-as-judge as a practice; it is not a complete halo theory, but it placed evaluator self-favouritism on the methodological agenda before Panickssery et al. measured it directly.
For practical evaluation, the implication is that exact-self records carry a default validity concern. They should be flagged at the analysis stage and excluded from headline general-quality claims unless a calibration argument is made on their behalf. The cost of exact-self elimination is small when the panel includes other judges; it becomes large only when exact-self is the only judge available, which is itself a sign that the evaluation cannot support a general-quality claim.
3.2 Same-provider preference
The evidence for same-provider preference beyond exact-self is weaker and more contested. Two strands matter.
The first is theoretical extension of exact-self results: if a judge can recognise the stylistic, structural, and policy fingerprints of its own outputs, then it likely recognises the fingerprints of sibling models from the same training lineage. Wataoka et al. 2024 analyse self-preference risk in LLM-as-judge and argue that the effect generalises to model-family-specific styles and policies, not only to the exact model. Pombal et al. 2026 extend the concern to rubric-based evaluation and argue that self-preference in this stronger sense includes outputs from the judge's model family, which is especially problematic for recursive self-improvement loops where in-family judging selects in-family generations.
The second is empirical: Stureborg et al. 2024, Large Language Models are Inconsistent and Biased Evaluators document a related "familiarity bias" — judges prefer text that is closer to their own output distribution, with lower perplexity under their own model, which would predict a same-provider preference that mostly survives blinding because style fingerprints persist after model names are removed.
These results justify treating same-provider-non-self as a distinct cell that should be reported separately. They do not justify a universal "same-provider preference = bias" conclusion, for a reason that should be stated bluntly: same-provider effects can also be capability-tracking signal. Modern LLMs differ systematically by provider on dimensions that any reasonable rubric will care about — reasoning style, refusal boundaries, citation discipline, verbosity, calibration, and instruction-following format. A same-provider judge preferring a same-provider output may be self-serving, but it may also reflect that the output better satisfies the rubric as that provider's models understand it. If the rubric is underspecified, this is one legitimate completion of the evaluation contract; if the rubric is precise, the same-provider preference is more clearly a validity threat.
Disentangling the two requires either human anchors or task instrumentation. Lacking those, the most defensible move is the one the literature converges on: report the cell separately, treat its size as a sensitivity signal, and avoid causal language about "bias" unless the controls are in place.
3.3 Cross-provider scoring
Cross-provider scoring is widely treated as the conservative analysis scope because it removes both author–judge contamination pathways. There is no body of work establishing that cross-provider scores are unbiased; the literature endorses them as a robustness view, not as a corrected truth. Three caveats matter.
First, dropping within-provider records changes the estimand. The cross-provider score answers a different question — what out-of-family judges think — and need not equal what an unbiased panel would have said. Second, cross-provider records can contain their own distortions: inter-provider style penalties, refusal-policy mismatch on edge cases, and domain-specific rubric interpretation differences. Third, filtering reduces sample size and can introduce provider-task imbalance, especially when authors or judges are not balanced across providers in the original design.
The right way to state cross-provider's role is: it is the scope most appropriate for general-quality claims that intend to generalise across organisations and judging systems, and the scope most resistant to the obvious "judge prefers its own family" criticism. It is not a debiased number, and it should not be presented as one.
3.4 The PsycheEval pilot
PsycheEval is a personality-conditioned LLM evaluation harness that explicitly separates exact-self from same-provider-non-self by including more than one model per provider in the judge panel. The v0.1 pilot used a tri-model panel for precisely this reason: a one-model-per-provider panel collapses exact-self and same-provider into a single cell, and the methodological question this article is about cannot be answered under that design.
In the 2026-04-26_micro_tri_model run, the analysis pipeline computes halo_audit_exact_self_minus_cross and halo_audit_same_provider_minus_cross per condition × author-model × rubric dimension. For GPT-5.5-authored outputs under condition C0 (the baseline condition with no auxiliary context), on the calibrated_challenge rubric dimension, the reported deltas are:
| Author model | Condition | Rubric dimension | Same-provider − cross | Exact-self − cross |
|---|---|---|---|---|
| gpt-5.5 | C0 | calibrated_challenge | +0.729 | +0.646 |
These are absolute deltas on a 1–5 scale. The single-cell observation supports two methodological points and refuses to support a third.
First, the cells separate. If exact-self were the only operative halo mechanism, the same-provider-non-self delta should approach zero after exact-self is excluded; it does not. This is the existence-proof point: same-provider effects are detectable beyond exact-self when the panel is designed to detect them, which justifies the cell-level taxonomy.
Second, in this specific cell, same-provider exceeds exact-self. That is interesting because it contradicts the naive ordering one might expect from a "closer relation means stronger halo" intuition. It is also a warning sign about that intuition: even if both effects are real, they need not be monotone in author–judge proximity. A single sibling judge can carry larger style affinity than the author's own self-recognition would, especially when self-recognition is weak or when the sibling's rubric interpretation is more rubric-aligned for that particular dimension.
Third — and this is the limit — the cell does not establish a stable magnitude. It is one rubric dimension, one author model, one condition, one run, with no uncertainty interval and no replication. The right way to use it in the methodology literature is as a demonstration of separability under tri-model panels, not as a population estimate. Anyone quoting +0.729 and +0.646 as parameters of LLM judging is over-extending the evidence. The right way to use it in the PsycheEval methodology itself is as a load-bearing reason to keep the multi-model-per-provider panel design, because the alternative cannot distinguish the two halo cells.
4. Inheriting vocabulary from human rater research
The terms "halo error," "leniency," "severity," and "central tendency" come from a century of human rater research, and they are useful for organising LLM-judge failure modes provided the import is treated as vocabulary, not as mechanism.
Thorndike, A Constant Error in Psychological Ratings (Journal of Applied Psychology, 1920) introduced halo error as the tendency for ratings on specific traits to be contaminated by a general impression of the ratee. The original observation was that intercorrelations between distinct trait ratings were too high to be explained by genuine inter-trait correlation; raters were letting global impressions colour specific judgments. The classical taxonomy expanded to include leniency (rater systematically rates above the scale midpoint), severity (rater systematically rates below), and central tendency (rater avoids extremes), each conceptualised as rater-level constants that distort observed scores away from the latent trait.
Saal, Downey, and Lahey 1980, Rating the Ratings: Assessing the Psychometric Quality of Rating Data (Psychological Bulletin) is the canonical caution against treating these as cleanly separable error sources. They argue that halo, leniency, and accuracy are conceptually and operationally entangled — corrections that reduce one statistic can mask shifts in another, and most operational "halo measures" do not isolate the construct they claim to. Their broader point is that rating error is a structural feature of the measurement system, not a nuisance term that can be subtracted out.
These results discipline the LLM-as-judge vocabulary in two ways. First, they justify the term halo for author–judge-relationship-conditional rating shifts, including the original sense of cross-dimension contamination ("the judge likes this author globally, so it rates every dimension up"). Second, they warn that naming a rater pattern is not the same as proving its causal mechanism — the same observed pattern can have multiple sources, and corrections can introduce their own distortions. The Saal/Downey/Lahey caution applies in full to LLM judges and is the main reason this article uses halo descriptively rather than as a synonym for bias.
Two things do not transfer from human-rating research to LLM-as-judge. The first is the assumption of a latent trait independent of the rater; for many LLM-as-judge tasks (helpfulness, style, persona fit) there is no rater-independent ground truth, so the human-rater framing of "true score plus rater error" does not apply cleanly. The second is the mechanistic story: human halo error is theorised in terms of impression formation and cognitive load, mechanisms LLM judges do not literally have. LLM rater shifts can mix style matching, reward-model inheritance, training-data overlap, instruction-hierarchy effects, learned evaluator conventions, and self-recognition. The analogy supplies the language; the mechanism has to be argued separately. See Construct Validity and Measurement Error Models for the broader framing.
5. Methodological contributions
The methodology stack for halo analysis has four layers. None of them is a correction in the strict sense. Each removes or surfaces a different source of evaluator dependence so the reader can see how much it matters.
5.1 Judge-stratified reporting
The cheapest and highest-information move is to report scores broken out by judge, not only aggregated. Judge-stratified tables expose three things at once: how much the aggregate depends on any one judge, whether judges agree on the ranking of authors, and whether one judge is driving an effect that disappears in the others. A benchmark that publishes "Model A wins 54.2%" without a per-judge breakdown is asking the reader to trust that the panel composition is irrelevant to the result.
Judge stratification is also a precondition for the other three methods. Without it, exact-self filtering and cross-provider filtering are bookkeeping operations the reader cannot audit.
5.2 Exact-self elimination at the analysis stage
Exact-self elimination is the simplest structural mitigation: drop records where author model and judge model are identical, then re-aggregate. It is appropriate as a default for benchmark headlines that claim general output quality. The cost is small when the panel includes other judges; it becomes prohibitive only when exact-self is the only judge, which is a sign that the evaluation cannot support the headline claim regardless.
Two failure modes are worth naming. First, asymmetric data loss: if each author model loses a different judge from its panel, the remaining judge set may differ in severity across authors, and the headline number can shift for reasons unrelated to the halo effect being mitigated. Second, predefined-rules drift: elimination should be defined before the data is collected, not selected after seeing the results. Post-hoc filtering of "the records that look bad" is not bias mitigation, it is degrees-of-freedom abuse.
5.3 Cross-provider filtering
Cross-provider filtering drops all within-provider records, leaving only judges with no provider affiliation to the author. It is the conservative scope for public benchmark claims because it is robust to the most obvious criticism — that in-family judges inflate in-family scores — and because it survives Panickssery-style self-recognition findings even under aggressive interpretation.
The cost is real and should not be obscured. Sample size drops, statistical power falls, the estimand shifts to "what out-of-family judges think," and inter-provider style preferences can introduce their own distortions in the opposite direction. Cross-provider filtering is the right primary scope for general-quality claims and the wrong primary scope for "how does our deployed judge view our deployed model" claims. The framing the literature converges to is that cross-provider is a cleaner estimand for some questions, not a corrected version of all of them.
5.4 Multi-provider, multi-model-per-provider panels
The structural design that makes the previous three methods possible is the panel itself. A panel must include at least two providers to support cross-provider analysis. It must include at least two models per provider to separate exact-self from same-provider-non-self. Without the second condition, every same-provider record is also exact-self, and the cells are not separable — the analysis can report cross-provider versus within-provider, but cannot disentangle the within-provider category into self and non-self.
The PsycheEval v0.1 tri-model panel was designed for this reason. It is not panel diversity for its own sake; it is the minimum structural commitment that lets the article's central taxonomy be empirically distinguished. A one-model-per-provider design — common in cheaper evaluations — collapses the analysis into a binary in/out-of-family cut and cannot answer the same questions.
Balance matters as much as count. If the panel has many GPT-family judges and one Claude judge, "cross-provider" effectively means "what the Claude judge thinks." The panel design should aim for enough records in each cell that the resulting estimates are not dominated by a single judge.
5.5 Reporting standard
The methods compose into a minimum acceptable reporting standard:
| Element | Why |
|---|---|
| Full-panel aggregate | Continuity with existing literature and a diagnostic baseline |
| Exact-self-excluded aggregate | Removes the most direct contamination pathway |
| Same-provider-non-self delta | Surfaces the contested cell separately |
| Cross-provider aggregate | Conservative scope for public claims |
| Per-judge breakdown | Lets the reader see who is driving the result |
| Sample sizes per cell after filtering | Detects power loss and provider imbalance |
| Uncertainty intervals | Prevents overclaiming small differences |
| Judge model identifiers and versions | Closed-judge drift is a known reproducibility issue |
A benchmark that publishes only the aggregate is making a claim it has not earned. A benchmark that publishes all four scopes with sample sizes is letting the reader see how much the headline depends on which records are included — which is the honest version of "halo control."
6. Implications for benchmark interpretation
The consequence for how to read existing benchmarks is direct. A single-provider judge panel — one provider's model judging a slate of authors that includes other models from the same provider — should not be used as strong evidence that in-family outputs are generally superior. The result may be correct, but it is conditional on the in-family judge, and there is no way to tell from the published number whether the conditional matters.
This is not a claim that single-provider judging is illegitimate. It is appropriate when:
- The question is internal product preference: "does our judge model like our author model?" The answer is then well-defined and the in-family panel is exactly the right instrument.
- The judge has been independently validated against human or task-grounded labels stratified by author model family, so the in-family halo has been measured and is small.
- The benchmark explicitly labels itself as panel-specific rather than as a general-quality measurement.
It is inappropriate when the benchmark advertises itself as cross-organisation general-quality measurement, makes claims that are intended to generalise across users and deployments, and has no validation against an external anchor.
A second implication concerns leaderboard movement. A model can climb a leaderboard either by getting better outputs or by drifting closer to the judge's style. Without judge stratification, the two are observationally identical in a single aggregate. Adding judges from a different provider, holding everything else fixed, is a stronger validity check than adding more authors. See Benchmark Overfitting and Goodhart's Law in AI Systems for the broader pattern.
A third implication is methodological economy. In some workflows the cheapest valid evaluation is a small cross-provider sample rather than a large single-provider one. The same is true the other way: if the question is product-internal, a large within-family sample answers it with less noise than a small cross-provider sample. The right design depends on the question, and stating the question is the first methodological act.
7. Second-order risks of de-haloing
Each of the four methods can introduce new distortions. The article frame so far has emphasised what de-haloing achieves; this section names what it can cost. The point is not to argue against mitigation — it is to argue against treating mitigation as automatically validity-improving.
Sample-size loss and shifted estimand. Exact-self exclusion and cross-provider filtering both reduce the population of records. The aggregate moves not because halo was removed but because the population changed. If the filter is large, confidence intervals widen sharply; if the filter is uneven across authors, providers, or tasks, the headline can shift for reasons unrelated to halo. Predefined rules and post-filter sample-size tables make this auditable.
Cross-provider style penalties. When the only retained judges are out-of-family, the panel may systematically penalise distinctive style choices that one provider's models happen to make and that no other provider's models recognise as rubric-aligned. This is not bias removal; it is the substitution of one set of evaluator preferences for another, with a different validity profile.
Calibration as bias laundering. A numerical "halo correction" trained on benchmark data can absorb known judge tendencies, but if trained on the same benchmark families, prompt strata, providers, or output styles, it can overfit and produce a more authoritative-looking number while encoding the same dependencies one layer deeper. The correction also creates maintenance debt: judge models, refusal policies, and rubric interpretations drift, and a correction trained on last year's panel may be wrong about this year's. The honest report names the correction as a model with its own assumptions, not as a transparent subtraction.
Panel aggregation hiding polarisation. Majority vote or mean across a multi-provider panel can produce a stable-looking aggregate while every individual judge is systematically biased in a different direction. A panel can mask provider-level disagreement that would have been useful information. Judge-stratified tables before aggregate claims are the corrective.
Over-correction. Eliminating same-provider records on the assumption that all such preference is bias can remove genuine capability-tracking signal. When the rubric is precise and the same-provider preference is invariant to controls for style, length, and refusal, this is a small concern. When the rubric is underspecified and the in-family preference reflects rubric interpretation, the elimination is a category mistake.
Provider taxonomy brittleness. "Provider" is not always well-defined. Hosted open-weight models, fine-tunes, distillations, model aliases, and judge snapshot identifiers all create edge cases where the in-family/cross-provider cut is judgment-dependent. The taxonomy should be defined operationally at the record level and the rules made explicit; otherwise the same dataset can produce different halo numbers depending on bookkeeping choices.
Public-leaderboard reading. Once a "halo-adjusted" number is published, downstream readers tend to treat it as the authoritative score, even when the adjustment is one robustness view among several. The safer publication pattern is to report multiple scopes side by side rather than to publish a single corrected ranking.
8. Active critiques
Three live disagreements run through this literature. None of them is fully resolved.
Is halo bias or signal? Same-provider preference can reflect favouritism (a validity threat) or rubric-aligned quality (a feature of the construct). Without human anchors or task-grounded ground truth, the two are not separable from the rating distribution alone. The strongest version of the critique is that "bias" framing presumes a rater-independent latent quality that for many LLM-as-judge constructs does not exist; what looks like halo may be one legitimate completion of an underspecified rubric. The weaker but more common version is that some same-provider preference is bias and some is signal, and the proportion is task-dependent.
The practical posture this article takes — and that the literature converges to — is to use causal language about bias only when the controls are in place: same-provider effects persist after style/length/refusal adjustment, do not reverse across prompt strata, and disagree with human anchors. Absent those controls, the safer term is halo-compatible effect or validity risk.
Does eliminating halo over-correct? A related critique is that aggressive structural elimination — dropping exact-self records, cross-provider filters, calibration adjustments — can over-correct by removing genuine signal along with the contamination. The cross-provider scope in particular is conservative for some questions and inappropriate for others; treating it as a universal default risks publishing numbers that no longer correspond to any deployment-relevant population. The counter-position is that for public benchmark headlines, the contamination risks are large enough that over-correction is a cheaper failure than under-correction. The disagreement is not over the existence of either failure mode; it is over which is the worse default.
Can halo be calibrated out? Calibration approaches — measurement-error correction, prediction-powered inference, learned debiasing layers — promise to recover the underlying construct from biased rater data given a small gold-labelled calibration set. The case for calibration is that it preserves sample size, can in principle recover the right estimand, and connects LLM-as-judge to a mature statistical literature on biased measurement. The case against is that calibration coefficients have not been shown to transport across judge versions, task strata, output styles, and provider mixes, and that an unstable correction is worse than no correction at all because it masks the instability. The honest current answer is that calibration is open: a viable research direction, not a settled methodology. Structural separation — stratified reporting and predefined exclusions — is more defensible until repeated multi-provider studies with human anchors show stable, transportable correction factors.
See Calibration, Measurement Error Models, and Prediction-Powered Inference for the broader statistical context; see Recursive Self-Improvement for why this matters for self-judging loops.
9. Calibration versus structural elimination
The choice between calibrating halo out and eliminating halo structurally is the operative methodological fork. The two strategies have different cost-validity profiles.
| Strategy | What it does | Cost | When it dominates |
|---|---|---|---|
| Structural elimination | Drops contaminated cells before aggregation. | Sample-size loss, shifted estimand, possible provider-style penalty. | Public benchmark headlines, cross-organisation claims, exploratory studies without human anchors. |
| Numerical calibration | Estimates judge error and corrects the aggregate. | Requires a calibration sample, transport assumptions, ongoing maintenance. | Settings with a stable human-anchor pipeline, where the correction has been validated across the relevant task and judge distribution. |
Both can fail in distinctive ways. Structural elimination can be honest about its costs (the dropped sample is visible) but cannot recover signal from the dropped cells. Numerical calibration can in principle recover signal but is more vulnerable to silent failure — a miscalibrated correction can produce an authoritative-looking number that is systematically wrong, and the failure mode is harder to diagnose than a sample-size table.
The current literature has more demonstrations of structural elimination's validity floor than of calibration's transport properties. Until that changes — until multiple studies show stable exact-self and same-provider coefficients across providers, tasks, judge versions, and style controls, with calibration improving agreement against external anchors rather than merely changing rankings — structural reporting is the more defensible default for public claims, and calibration is more defensible for product-internal monitoring where the calibration set is large and current.
The two are not mutually exclusive. A serious report can publish structural-elimination scopes for the headline and provide calibration estimates as a sensitivity view. The reverse — calibrated headline with a structural-elimination footnote — concentrates more inferential weight on a methodology whose transport properties are still open.
10. Relevance to PsycheEval
PsycheEval's load-bearing relationship to this methodology is design-level, not narrative-level. The v0.1 panel is tri-model — at minimum two providers and at least two models within one provider — for a single concrete reason: a one-model-per-provider panel cannot separate exact-self from same-provider-non-self. The cell-level taxonomy this article is built around is not optional decoration for the PsycheEval design; it is the analysis the design exists to support.
The pilot demonstration in 2026-04-26_micro_tri_model makes the point concretely. For GPT-5.5-authored outputs in condition C0 on the calibrated_challenge rubric dimension, the same-provider-non-self delta of +0.729 exceeds the exact-self delta of +0.646. The two halo cells are separated; the ordering between them is not the textbook intuition; the panel design is what makes either observation possible.
What the cell does not establish bears restating. It is a single cell, in a single run, without uncertainty intervals or replication, on a rubric dimension that itself has known sensitivity to style. The methodological lesson is solid; the magnitude estimate is fragile. The right way for downstream PsycheEval methodology to use it is as a demonstration of separability — the reason the tri-model panel is non-negotiable — not as a calibration coefficient for any subsequent benchmark claim.
What would move the evidence from suggestive to robust is a balanced mixed-effects reanalysis of the existing records: random effects for prompt/scenario, fixed indicators for exact-self and same-provider-non-self, judge severity effects, style/length/refusal controls, and a human-anchor subset on the records where the provider-linked coefficients are largest. The decisive result would not be a non-zero coefficient on either halo cell; it would be whether same-provider-non-self remains large after exact-self removal, style controls, and judge severity adjustment, and whether removing or calibrating it improves agreement with an external anchor rather than merely changing the ranking of authors. Until that analysis exists, PsycheEval halo numbers should be reported with the same stratified scope structure this article recommends for any benchmark: full panel, exact-self-excluded, same-provider-non-self delta, cross-provider, with sample sizes and uncertainty for each.
See PsycheEval for the broader harness, Calibrated Challenge for the specific rubric dimension on which the pilot cell sits, and Persona Conditioning for the conditions (C0, C1, …) under which the records are produced.
11. The open question
The single most important unresolved question in this literature is whether halo can be calibrated out — debiased by statistical correction — or whether it requires structural elimination — dropping contaminated cells before aggregation. The two strategies have different sample-size, transport, and audit properties, and the empirical case for either as a universal default is not yet settled.
The closely related second question is whether same-provider preference is mostly bias or mostly signal, and whether the proportion is stable enough across tasks, judges, and prompt strata to support a generic correction. The honest answer is that the proportion is task-dependent and that the studies that would establish stability have not been done at the scale and balance required.
The methodology this article recommends survives both unresolved questions because it does not depend on them. Stratified reporting and predefined structural exclusions are robust to "halo is bias" and "halo is signal" — they let the reader see how much the answer depends on which records are included, and they let downstream calibration work be added as a sensitivity view rather than substituted for the underlying numbers.
The reasonable expectation is that the field will accumulate calibration evidence gradually, that some tasks and judge populations will admit stable corrections while others will not, and that the default for public claims will remain structural for years. The faster the multi-provider, multi-model-per-provider, human-anchored studies appear, the faster calibration becomes a legitimate default. The cost of waiting is that benchmark headlines that depend on calibration coefficients with unknown transport properties are headlines that may not survive the next judge model update.
Companion entries
Core theory: LLM-as-Judge, LLM Evaluation, Construct Validity, Measurement Error Models, Calibration, Prediction-Powered Inference, Human Preference Modeling
Halo and related rater effects: Self-Preference Bias, Model-Family Preference, Familiarity Bias, Position Bias, Verbosity Bias, Halo Error, Leniency and Severity, Central Tendency
Methodology: Judge-Stratified Reporting, Cross-Provider Filtering, Multi-Provider Judge Panels, Exact-Self Elimination, Mixed-Effects Models for Rating Data, Calibration Sets, Human-in-the-Loop Evaluation
Empirical reference points: MT-Bench, Chatbot Arena, JudgeBench, AlpacaEval, Prometheus Evaluator, PsycheEval, Calibrated Challenge
Counterarguments and futures: Halo as Capability-Tracking Signal, Over-Correction in Eval Adjustments, Calibration Transport Failure, Recursive Self-Improvement, Synthetic Data Feedback Loops, Evals as Instrumentation