Abstract. Synthetic-first evaluation is the design stance of running an end-to-end evaluation pipeline — synthetic personas, synthetic scenarios, synthetic judges, full analysis code — before recruiting a single human participant or rater. The empirical motivation is asymmetric cost: rubric ambiguity, judge bias structure, condition coverage gaps, length confounds, position effects, and analysis-pipeline bugs all manifest in synthetic data at orders-of-magnitude lower cost than in a human study, and they will manifest there whether the evaluator looks for them or not. The methodological claim is narrower than it appears: synthetic pilots can falsify and debug an evaluation harness, but they cannot validate a treatment for humans. The line between "the harness now measures what it claims to measure on synthetic data" and "the intervention works on real users" is the line between Methodology Pilot and Product Validation, and it does not dissolve under more synthetic effort.
Coverage note: verified through May 2026.
Definition
Synthetic-first evaluation is a sequencing doctrine for eval programs, not a replacement methodology for human studies. The doctrine has three parts.
The first part is a procedural claim about workflow ordering. Before any human is asked to interact with the system under test, before any rater is recruited to label outputs, before any IRB protocol is finalized, the evaluation pipeline is run end-to-end on synthetic data. Synthetic personas play the role of the target population; synthetic scenarios play the role of the field encounters; synthetic judges play the role of the human raters; the same analysis code, randomization scheme, and reporting templates that would be used on the real study are exercised on the synthetic data. The pipeline is not a sketch. It is the actual instrument.
The second part is an empirical claim about cost asymmetry. Most measurement-design failures are detectable on synthetic data — not because synthetic data resembles humans, but because most measurement-design failures are structural defects in the harness rather than facts about the target population. A pairwise judge with a 20-point position-slot preference will exhibit that preference on synthetic transcripts. An analysis script that mishandles paired-difference variance will mishandle it on synthetic outputs. A rubric whose four dimensions collapse to a single fluency factor will collapse them under any judge with a fluency prior, human or synthetic. The synthetic run does not certify that the harness works on humans; it certifies that the harness is not already broken in ways visible without humans.
The third part is a discipline about claims. Synthetic-first does not produce evidence about treatment efficacy, about ecological validity, about real-user effect size, or about deployed fairness. It produces evidence about the measurement instrument. The vocabulary needed to keep these straight — methodology pilot versus product validation, harness development versus efficacy claim, bias detection versus treatment validation — is the article's central object.
The pattern has analogues in other disciplines that have practiced simulation-before-experiment for decades. Statistical methodologists run Monte Carlo simulations against known data-generating processes before recommending an estimator (Morris, White & Crowther 2019). Education researchers prototype experimental protocols against simulated participants to find design defects before recruiting students (de Leeuw, Gilbert & Luchterhandt 2023). Clinical trial designers run simulated patient cohorts to estimate power and find boundary cases. What is new in AI evaluation is not the synthetic-first stance — it is that the simulator now produces text that looks like the target output, which makes it tempting to mistake the simulator for the target.
The Empirical Motivation
The argument for synthetic-first is not aesthetic. It is built on a base rate.
The base rate from evaluation work is that first-pass instruments are flawed. Rubrics show ceiling effects. Prompts drift from their schemas across runs. Pairwise designs contain ordering effects that the designer did not anticipate. Scenario sets miss adversarial cases. Analysis scripts encode assumptions — paired versus unpaired comparisons, clustering structure, exclusion rules — that the designer did not consciously make. Public benchmark research is littered with examples: SWE-Bench+ found solution leakage and weak test cases that, after filtering, dropped reported success rates substantially (SWE-Bench+); EvalPlus showed that augmenting HumanEval's test coverage produced pass@k drops of 19.3–28.9%, demonstrating how thin the original tests had been (EvalPlus). These are public, peer-reviewed instruments. Private eval suites — usually smaller, less peer-reviewed, written by the same team incentivized to ship — have no reason to be cleaner. The base rate of broken-on-first-pass is high.
The cost of discovering these defects late is the load-bearing fact. Recruiting human raters or human participants is expensive in money, time, and (in many domains) ethical authorization. Once recruitment has happened, the choice is between using flawed data, re-recruiting after a redesign, or quietly continuing and publishing the original protocol. The least defensible option is the most tempting.
Synthetic pilots compress the discovery loop. A pairwise judge can be run against 500 synthetic transcripts in minutes. A rubric's dimension separability can be tested by perturbing one dimension at a time and watching whether judge scores move only on that dimension. A length confound can be checked by generating matched-length variants of the same content and seeing whether the score follows length or follows substance. None of these checks requires a single human. All of them can fail in ways that would have invalidated a human study post hoc.
This is the strongest version of the empirical motivation. The weaker, more common version is "synthetic pilots are cheap, therefore we should do them." The stronger version is: the failure modes that synthetic pilots reliably surface are the same failure modes that, if undetected, will compromise the human study. Discovering them with a thousand synthetic transcripts is not a substitute for the human study; it is a precondition for the human study being worth running.
Vocabulary: Methodology Pilot vs Validation Claim
The discipline of synthetic-first depends on a vocabulary that keeps claim types separable. The vocabulary has four pairs.
Methodology pilot versus product validation. A methodology pilot is a study of the measurement instrument: does the harness measure what it claims to measure, is it sensitive to obvious confounds, does the analysis pipeline recover known effects when known effects are injected? A product validation is a study of the intervention under test: does the feature produce the claimed benefit in the target population? The two studies share infrastructure but answer different questions. A synthetic-first program does the first kind. A human study, after the harness is debugged, does the second kind. Confusing the two is the first failure mode of the discipline, and most of the others descend from it.
Harness development versus efficacy claim. The harness is the evaluation instrument — prompts, scenarios, rubric, judges, randomization, aggregation, analysis. The efficacy claim is "the intervention improves outcomes in the target population." A harness can be excellent and the intervention can still fail; an intervention can be effective and the harness can still produce noisy or biased estimates. Synthetic-first is a harness-development practice. It does not produce efficacy claims, and the article wishing to use it for efficacy is in the position of a metallurgist using their own caliper to verify that the caliper is well-calibrated.
Synthetic-can-detect-bias-but-not-validate-treatment. Synthetic pilots can detect certain kinds of bias: judge position bias, same-provider halo, length sensitivity, channel disagreement, rubric collapse, condition-coverage gaps, generator artifacts. They cannot validate that the intervention is unbiased in deployment. The synthetic population does not represent any real population unless it has been externally validated against one, and the validation step is exactly the kind of human work the doctrine is trying to defer. A clean synthetic run is bounded evidence about the harness, not evidence about deployment fairness.
Synthetic-first versus synthetic-final. Synthetic-first is the doctrine that synthetic runs precede human studies. Synthetic-final is the (incorrect) doctrine that synthetic runs replace human studies. The first is a useful sequencing discipline. The second is a methodological mistake that happens when the synthetic harness becomes good enough that running the human study feels redundant. The cost of synthetic-final is the cost of every evaluation program that mistook a polished benchmark for evidence about people.
These pairs are not stylistic. They are the operational machinery that prevents synthetic pilots from being laundered into efficacy claims. The wording matters because the absence of the wording is the failure mode: an article that describes synthetic results without committing to one side of each pair has already drifted toward the more aggressive claim.
The Simulation-Before-Experiment Tradition
Synthetic-first has the disadvantage that the term is new and the advantage that the practice is old. Three traditions feed it.
Monte Carlo and the ADEMP Frame
Statistical methodologists have run simulation studies for decades because the alternative — closed-form analysis of estimator behavior under realistic data-generating processes — is intractable for most modern methods. Morris, White, and Crowther codified the practice with the ADEMP framework: define the Aims, the Data-generating mechanism, the Estimands, the Methods being compared, and the Performance measures, in that order, before running anything (Morris, White & Crowther 2019). The discipline is rigorous because the truth is known: the simulation's data-generating process is chosen by the analyst, so the estimator's behavior can be compared against a ground truth that does not exist in field data.
The lesson that transfers to AI evaluation is the structure, not the literal practice. ADEMP works because the analyst commits to the data-generating mechanism before drawing inferences. A synthetic-first AI eval that doesn't commit to its data-generating mechanism — that lets the synthetic persona distribution, the scenario coverage, the judge panel, and the rubric all be tuned post hoc against observed results — is doing something closer to exploratory fitting than to simulation. The "synthetic" label by itself is not the methodology; the registered design is.
The lesson that does not transfer is the strength of the ground-truth claim. In statistical Monte Carlo, the analyst knows the true value of the estimand because the analyst chose it. In synthetic AI evaluation, the analyst does not know what real users want, do, or experience. The synthetic data has a generating process — but the generating process is an LLM with its own priors, not a model of the target population validated against the target population. The estimand of interest (real-user benefit) is not in the synthetic world.
Simulated Participants in Education and Clinical Research
Education researchers and clinical educators have used simulated participants for decades to debug experimental designs and train practitioners. De Leeuw and colleagues extended this practice into computational form by using language models as simulated participants for cognitive psychology experiments, arguing that simulation can reveal protocol defects, instruction ambiguity, and interface bugs before behavioral data collection (de Leeuw, Gilbert & Luchterhandt 2023). The methodology is explicitly framed as preparation, not substitution: the simulated participants reveal failures that would have wasted real participants' time, and the design then proceeds to recruitment with higher confidence that the protocol is interpretable.
Clinical research has a longer history of simulated patients in protocol design and provider training, with the same epistemic structure: the simulation can falsify procedural designs but cannot establish clinical outcomes. The Society for Simulation in Healthcare and adjacent traditions treat simulation as a category of preparation, not a category of evidence about patient outcomes.
The unifying claim across these traditions is the same: simulation is valuable because it is controlled, inspectable, and cheap; it is weak for the same reason, because the analyst chose what to simulate. Joshua Epstein's "generative social science" framing makes the philosophical version of the point explicit — explaining a phenomenon by generating it from interacting agents is a sufficient condition for a candidate explanation, not a proof that the candidate explanation is the operative one in the world (Epstein 2006).
LLM-Simulated Survey Respondents
The most directly cited precedent in AI evaluation is Argyle, Busby, Fulda, Gubler, Rytting, and Wingate's "Out of One, Many," which argued that GPT-3 conditioned on demographic backstories could reproduce aggregate survey-response patterns from human samples with sufficient fidelity to support cautious use as a methodology proxy (Argyle et al. 2023). The paper introduced the term "algorithmic fidelity" to describe the condition under which LLM-simulated subjects can stand in for human subjects in bounded social-science applications — replicating known aggregate patterns, exploring counterfactual treatment frames, and stress-testing survey instruments.
The paper is widely cited as evidence that LLMs can simulate humans. The paper itself is narrower than that reading. Argyle and colleagues argue for algorithmic fidelity under demographic conditioning for aggregate-distribution behavior. They do not argue for individual-level substitution. They do not argue that simulated subjects validate new interventions in unobserved domains. They do not argue that LLM-generated responses can replace human samples for treatment-effect estimation in deployed systems.
Subsequent work is an active series of warnings. Santurkar and colleagues found substantial demographic misalignment between LM-expressed opinions and the opinions of the demographic groups the models were prompted to represent (Santurkar et al. 2023). Dominguez-Olmedo and colleagues documented ordering and labeling biases in language models' survey responses that have no analogue in human respondents, undermining the literal interpretation of LLM survey behavior (Dominguez-Olmedo et al. 2023). Tjuatja and colleagues showed that LLM response biases often fail to match the response biases human subjects exhibit in survey-design experiments, which means the simulated population diverges from real populations exactly where survey methodology cares most about the divergence (Tjuatja et al. 2024). Gui and Toubia gave the causal-inference version of the critique: simulated subjects can violate the unconfoundedness assumption that licenses causal inference, because conditioning on a treatment can shift unobserved variables the analyst did not specify (Gui & Toubia 2023).
The honest reading is that Argyle and colleagues opened a door for bounded methodology use of LLM-simulated respondents, and the follow-on literature has been spending the time since walking the boundary back to where it should have been all along. Synthetic-first evaluation lives inside the door but has to keep noticing where the boundary is.
What Synthetic Pilots Can Detect
The argument for synthetic-first rests on a specific list of measurement-design failures that synthetic data reliably surfaces. The list is concrete enough to be a checklist.
Judge Halo and Dimension Separability
A rubric typically claims that several distinct quality dimensions are being measured: helpfulness, calibration, safety, faithfulness, voice, agency support, sycophancy resistance, and so on. The harness's implicit claim is that the judge can assign different scores to these dimensions in response to outputs that vary along one dimension but not others. The actual behavior of LLM judges is often dominated by a single fluency-or-quality factor, with the named dimensions correlated near unity. This is the classical halo effect transposed to LLM judges.
Halo separability is testable synthetically. Generate transcripts that deliberately vary along one rubric dimension (say, calibration) while holding others constant (helpfulness, voice). Score them. If the judge's calibration score moves but the other scores don't move, the dimension is operationally separable. If all scores move together, the dimension is decorative — the judge is producing a single signal labeled four ways. PsycheEval's v0.1 pilot caught judge halo precisely this way, before any human rater would have spent time on a four-dimension rubric that was really a one-dimension instrument.
The mitigation is not "use a different judge until halo goes away." Often no judge will pass cleanly, and that is the finding: the rubric needs different language, the dimensions need different anchor examples, or the dimensions need to be collapsed honestly. The synthetic pilot has revealed something about the measurement instrument that the human study would have revealed at twenty times the cost.
Position Bias and Counterbalancing
Pairwise judging is the dominant scoring mode for many LLM evaluations: present output A and output B, ask which is better, aggregate over many pairs. The implicit assumption is that the judge's preference reflects the substance of A versus B, not the position of A versus B in the prompt. The actual behavior of LLM judges is that position is often a substantial confound. Position bias has been measured across 15 judge models and more than 150,000 evaluation instances and confirmed as a general property of LLM-as-judge rather than a model-specific quirk (Position Bias in LLM Judges).
Position bias is detectable synthetically by counterbalancing: present each pair twice, once as AB and once as BA. If the judge prefers slot-A or slot-B at a rate detectably different from 50% on identical content, the bias is real and must be controlled for. PsycheEval's v0.2 hard pilot ran exactly this audit and found judge-family-specific slot-B preferences ranging from approximately zero to +0.31 across tested judge-pair combinations. The implication is methodologically important: pairwise verdicts from a single judge without counterbalancing are confounded by an effect that can be larger than many real treatment effects, and the only way to know whether the verdict survives the confound is to counterbalance.
The synthetic pilot does not certify that no position bias exists in human raters — but human raters' position biases have been studied for decades, and the design controls for them are standard. The synthetic pilot certifies that the automated harness, the one that will produce many of the scores in any modern evaluation, is not silently encoding a position preference into the verdict.
Channel Disagreement
Many evaluations score outputs through more than one channel: self-report from the simulated user, behavioral trace counts, free-text feedback summaries, scalar-rubric judges, pairwise-preference judges. The implicit assumption is that these channels are pointing at the same construct from different angles. The actual behavior is often that the channels disagree systematically, and the disagreement is the methodologically interesting finding.
PsycheEval surfaced channel disagreement directly: scalar judgments of a rubric dimension did not correlate cleanly with pairwise preferences on the same outputs, and free-text feedback sometimes pointed to features the scalar rubric did not score. The methodological consequence is that aggregating across channels can hide opposite directional effects; the methodological opportunity is that channel disagreement is diagnostic information about what the construct actually means.
The synthetic pilot is the right place to find this. If the channels disagree on synthetic data, they will disagree on human data, and the harness needs to decide in advance how to interpret disagreement. If the team waits to discover disagreement in a human study, the only options are to publish the disagreement as a finding, to pick one channel post hoc, or to invent a reconciliation rule after seeing the data. None of these options is methodologically clean.
Length Confounds
LLM judges have been documented to reward longer outputs even when length is not the substantive variable (MT-Bench / Chatbot Arena). The implication for evaluation is that any treatment that affects output length is confounded with whatever-the-judge-prefers, and any pairwise verdict between treatments with different length profiles is at least partially driven by length rather than substance.
Length confounds are trivially detectable synthetically: generate matched-length variants of the same substantive content, run them through the judge, and see whether the score follows substance or length. If substance is real, length-matched scoring should preserve the substantive ordering. If length is dominant, length-matched scoring should collapse the apparent effect.
PsycheEval ran exactly this audit and produced one of the more methodologically important results in its hard pilot: the C4 condition beat the C1_padded condition (matched-length control) at 62.3%, demonstrating that the apparent C4-versus-C1 advantage was not solely a length artifact, while several other apparent advantages collapsed under length control. The methodological win is not "C4 is the right condition." The methodological win is "we know which apparent advantages are substantive and which are length, and the rubric for accepting future condition comparisons now includes length controls as a precondition."
Condition Coverage and Pipeline Bugs
The remaining synthetic-pilot detection targets are less glamorous and equally important. Condition coverage gaps — combinations of factor levels that the design intends to test but that the prompt generator silently fails to produce — are detectable by counting the cases. Pipeline bugs — analysis scripts that mishandle paired comparisons, clustering structure, missing data, or exclusion rules — are detectable by injecting known effects into synthetic data and asking whether the pipeline recovers them. Negative controls — conditions that should produce null results by design — are detectable by checking whether the harness produces "significant" effects under the null. Each of these is a structural defect in the harness, and each manifests in synthetic data because it manifests in any data that passes through the harness.
The cumulative picture is that synthetic-first is not a single technique. It is a checklist of detection targets, each with a specific test, each cheap to run, each capable of invalidating a human study post hoc if missed.
PsycheEval as a Worked Example
PsycheEval is a personality-conditioned LLM assistant evaluation program whose published pilot reports illustrate the synthetic-first stance in operating form. The v0.1 report described itself explicitly as a methods pilot, not as evidence about real users, and the v0.2 hard pilot extended the methodology with AB/BA counterbalancing and length controls before any human-rater work began. The PsycheEval framing is worth reading closely because it shows what the discipline looks like when it is honored, and what specific findings the synthetic-first stance produced.
The v0.1 Methods Pilot
The v0.1 pilot was a small synthetic study of personality-conditioned vs unconditioned assistant outputs, scored by an LLM-judge panel on a multi-dimension rubric. The headline of the report was not a treatment-effect estimate. The headline was a list of measurement-design failures the pilot had surfaced: judge halo across rubric dimensions, channel disagreement between scalar and pairwise judges, and risk of length confounding for the conditions where the personality conditioning produced longer outputs. The published framing was deliberate: this is a methods pilot, the synthetic data cannot validate the intervention, and the next step is harness revision before any human study.
The methodological discipline this represents is not unusual in principle. It is unusual in practice. Many AI-evaluation pilot reports headline an apparent effect and treat the measurement-design failures as caveats. PsycheEval's v0.1 inverted the prominence: the measurement-design failures were the headline, and the apparent effects were the caveat. That inversion is the operational test of whether a team is doing synthetic-first or doing synthetic-final with rhetorical hedging.
The v0.2 Hard Pilot
The v0.2 hard pilot extended the methodology to address the v0.1 findings. The key additions were AB/BA position counterbalancing for all pairwise comparisons, length-matched control conditions (notably the C1_padded condition designed to test whether C4's advantage was length-driven), and a wider judge panel to estimate judge-family effects.
The numerical findings, treating each as a methodology result rather than an efficacy claim:
| Comparison | Counterbalanced verdict | Methodology reading |
|---|---|---|
| C5_CONTRACT vs C5 | C5_CONTRACT wins 67.7% | Contract-form prompt produces a measurable advantage that survives slot counterbalancing |
| C4 vs C5 | C4 wins 68.0% | C4 outperforms C5 under counterbalancing, suggesting the conditioning approach matters beyond the contract framing |
| C4 vs C1_padded | C4 wins 62.3% | The C4 advantage survives length-matching against a padded C1, so the effect is not solely a length artifact |
| Slot-B preference (across judge families) | Approximately 0 to +0.31 | Judge-family-specific position bias is non-trivial and varies across pairs; pairwise verdicts without counterbalancing are confounded |
The most important number in this table is the last one. A position bias of +0.31 in slot-B preference is comparable in magnitude to many treatment effects of interest in LLM evaluation. Any pairwise verdict from a judge family with that level of position sensitivity, run without counterbalancing, is producing a score that is approximately as much about position as it is about substance. The v0.2 pilot's discovery is not that one condition won — it is that the harness, in its un-counterbalanced form, could not reliably attribute apparent wins to substance versus position.
The published claim discipline is correspondingly tight. The v0.2 report does not say that personality-conditioned assistants improve outcomes for real users. It says that, after counterbalancing and length controls, certain condition comparisons survive as harness-level findings and that others collapse. The validation step — whether any of this matters to humans, and at what effect size — is explicitly deferred to a planned human study.
What the PsycheEval Case Demonstrates
The case demonstrates three things worth naming explicitly.
First, the synthetic-first stance is operational, not aspirational. It is a workflow with specific detection targets, specific controls, and specific report-format discipline. A team can do it or not do it. PsycheEval's pilots are evidence that doing it produces methodologically informative results before any human is recruited.
Second, the synthetic findings constrained future claims. The v0.1 finding of judge halo forced rubric revisions before the v0.2 pilot. The v0.2 finding of slot-B bias forced counterbalancing as a non-optional control. The v0.2 finding that the C4-vs-C1 effect survived length matching constrained which apparent advantages the team could later treat as candidates for human-rater testing. Each finding moved the harness in a direction the team's prior beliefs would not have moved it.
Third, the case is not evidence that PsycheEval works as a deployed assistant. The case is evidence that the evaluation harness for PsycheEval is methodologically defensible after two rounds of synthetic auditing. The validation question — whether real users benefit, whether real raters agree with the LLM-judge verdicts, whether the synthetic-tuned harness generalizes to humans — remains open and is correctly framed as a separate study.
The Construct-Validity Caveat
The most attractive version of the synthetic-first claim is that, after the synthetic pilot has caught the rubric problems, the judge bias structure, the condition coverage gaps, and the analysis-pipeline bugs, ecological validity is the only remaining question for the human study. That version is too narrow.
Ecological validity — whether results from a controlled study generalize to the deployed setting — is one of several validity questions that human contact can settle. The others are at least as important.
Construct validity is the question of whether the measurement is measuring what it claims to measure. A rubric whose four dimensions are operationally separable under synthetic perturbation may still be measuring the wrong construct, in the sense that the construct is not what humans value, recognize, or experience. Real users may reward a different conversational move than the rubric scores. Real raters may interpret a rubric item differently from the synthetic judges' interpretation. The synthetic pilot cannot detect this kind of failure because the synthetic pipeline is internally consistent: the judges and the rubric were designed together, and they agree on what the construct means.
Criterion validity is the question of whether the measure predicts an outcome that matters. A synthetic-validated rubric for "sycophancy resistance" cannot be evaluated for criterion validity against, say, long-term user trust, because long-term user trust is not in the synthetic world. Criterion validity is an empirical relationship between a measurement and an external outcome; both ends of the relationship have to be measured on the same participants.
Comprehension and motivation effects. Real participants may misunderstand a scenario, optimize for social desirability, disengage, respond under stress, experience fatigue, or interpret a prompt through lived context the generator never modeled. These are not noise around an otherwise correct measurement. They can change what the measurement measures. A rubric that produces clean separation under synthetic personas may produce different separation under humans because the humans interpret the items differently.
Heterogeneity of treatment effect. Synthetic personas, even when diverse, are diverse along the dimensions the generator can articulate. Real populations are diverse along dimensions that include unobserved confounders, subgroup-specific responsiveness, and rare-but-important cases that the synthetic distribution underweights or misses entirely. An intervention with a large average effect in the synthetic world may have a small or negative effect in a specific subgroup that matters for deployment.
The honest framing of the human study, after a clean synthetic pilot, is therefore not "test ecological validity." It is "test construct validity, criterion validity, comprehension, motivation, heterogeneity, and ecological validity, with effect-size estimation as the headline deliverable." Synthetic-first removes a set of upstream failures that would have made the human study uninterpretable. It does not narrow the human study to a single remaining question.
Generator and Judge Self-Confirmation
A synthetic-first pipeline that uses an LLM to generate personas, an LLM to generate scenarios, an LLM to author candidate outputs, and an LLM to judge those outputs has a structural problem the discipline must address directly. Even with different vendors, the system is sampling from overlapping text-trained priors, instruction-following norms, and aesthetic preferences. The harness can optimize for what LLMs think a good response to a simulated user looks like. That optimization is informative about what LLMs reward; it is not yet evidence that humans experience the response as helpful, challenging, safe, or trustworthy.
The mitigation is not "use a more diverse judge panel until the problem goes away." Diverse judge panels reduce single-model artifacts but do not break the underlying loop, because the models in the panel share training-data ancestry and reward-modeling lineage. The mitigation is to treat the synthetic harness as a measurement instrument whose own biases are part of the object under study, and to design audits whose results reveal those biases rather than launder them.
Concretely, this means: separating persona generation from judging by vendor when possible; including human anchor examples in the judge calibration set; running negative controls where the harness should produce null results; checking that effects do not concentrate in a single persona family; treating channel disagreement as a substantive signal rather than noise to average over; and writing into the protocol, in advance, which kinds of synthetic findings are allowed to revise the harness and which must be held for human confirmation.
The strongest version of this concern is sometimes raised as an objection to synthetic-first as a whole: that the discipline is a sophisticated form of self-confirmation, in which the team builds a system that LLMs reward, and then reports the LLM reward as evidence. The objection is correct as a description of the failure mode and incorrect as a description of the discipline. The failure mode is real and avoidable. The discipline's machinery — counterbalancing, length controls, dimension-separability audits, condition coverage maps, negative controls, claim-boundary vocabulary — exists exactly to prevent the failure mode. A team that does the audits and respects the boundary is not doing self-confirmation; it is doing measurement engineering. A team that skips the audits and reports the LLM-judge scores as if they were human evidence is doing self-confirmation under the flag of methodology.
The audit machinery, in other words, is the difference between synthetic-first and synthetic-decoration.
Red-Teaming as a Cousin Discipline
The closest existing cousin to synthetic-first evaluation is red-teaming as a safety practice. The two share an epistemic structure that is worth making explicit because the structure has implications for how the article should treat both.
Red-teaming as practiced by frontier labs and codified in standards documents is the deliberate, structured probing of an AI system for failures: capability surprises, jailbreaks, harmful outputs, policy violations, manipulation patterns. NIST's Generative AI Profile under the AI Risk Management Framework describes red-teaming as a controlled stress-testing practice for safeguards, framed as one element of a broader safety evaluation portfolio that includes field testing and direct human feedback (NIST AI 600-1 GAI Profile). OpenAI's own red-teaming documentation makes the asymmetry explicit: red-teaming is not a complete solution; it is one practice among several, and a clean red-team pass is not evidence of safety.
The epistemic structure red-teaming and synthetic-first share is asymmetric evidence. A red team that finds a failure has produced direct, falsifying evidence that the system has the failure. A red team that does not find a failure has produced bounded evidence that the system is robust against the kinds of probes the red team tried. The first kind of evidence is strong. The second kind is weak. Symmetric framing of red-team results — "we red-teamed and found nothing, therefore the system is safe" — treats the second kind as if it were the first, and is the most common misuse of red-teaming in practice.
Synthetic-first evaluation inherits the same structure. A synthetic pilot that detects judge halo, position bias, length confounds, or pipeline bugs has produced direct falsifying evidence about the harness. A synthetic pilot that detects none of these has produced bounded evidence that the harness is robust against the kinds of audits the team thought to run. The first kind of evidence is what synthetic-first is for. The second kind is what gets oversold when the discipline drifts toward synthetic-final.
The companion implication is that red-teaming can be framed as a synthetic-first stress test of evaluation harnesses, not just of production systems. Adversarial synthetic personas, edge-case scenarios, deliberately confounded conditions, and stress-test rubric items all serve as red-team probes of the harness. The methodology benefit is the same: cheap discovery of failures that would have been expensive to discover later. The epistemic limit is also the same: absence of failure is not validation.
Lifecycle Design: Synthetic-First, Not Synthetic-Final
The article's recommended path is staged. The stages are not optional; the order is not optional; and the gates between stages exist to prevent the discipline from collapsing into synthetic-final.
| Design | Strength | Failure mode | Best use |
|---|---|---|---|
| Human-first pilot | Immediate contact with real participant behavior | Expensive debugging; small samples confound harness bugs with human effects; ethical exposure for participants of a flawed instrument | When the construct is embodied, contextual, or stakes-sensitive in ways that simulation cannot capture at all |
| Synthetic-only evaluation | Fast, broad, cheap, reproducible | Self-confirming simulation; generator and judge biases mistaken for evidence about people; claim laundering | Internal harness audits and red-team stress tests only; never a deployment-licensing instrument |
| Synthetic-first, human-validated lifecycle | Separates cheap methodology debugging from expensive validation | Can delay validation indefinitely if gates are vague or endlessly tuned; preregistration discipline is non-optional | Default for complex AI evaluation programs where the construct is partially simulable and human validation is non-negotiable |
The recommended lifecycle, in operational form:
Step 1. Define the construct, estimands, target population, and forbidden claims before generating any synthetic data. The pre-registration discipline matters because everything downstream becomes evidence of the registered question or off-protocol exploration. The list of forbidden claims is as important as the list of estimands: write down which conclusions the methodology pilot cannot produce, so that the report cannot drift toward them.
Step 2. Generate synthetic personas and scenarios for coverage. Include ordinary cases, adversarial cases, negative controls, and known-confound probes. The synthetic distribution is not the target population; it is a coverage map of the harness's tested input space. Document the gaps explicitly.
Step 3. Run multiple generators and judges where feasible, separating persona generation from judging across vendors when possible. Single-model harnesses are easier to build and harder to trust. Multi-model panels reduce single-model artifacts but do not break the underlying training-data overlap, so the audit discipline still applies.
Step 4. Use synthetic results to revise the harness until it passes explicit readiness checks. Rubric dimensions are operationally separable under perturbation. Judges are not dominated by length or position. Condition coverage is balanced. The analysis pipeline recovers injected effects. Negative controls return null. Channel disagreements are predeclared as either substantive or as failure conditions. Each readiness check is a falsifiable test, not an aesthetic standard.
Step 5. Freeze the human-study protocol after the methodology pilot. Any subsequent harness change before the human data is collected counts as a new pilot cycle and is documented as such. The discipline of the freeze is what separates synthetic-first from rolling exploratory tuning.
Step 6. Run the human validation study. Test construct validity, criterion validity, comprehension, motivation, heterogeneity, ecological validity, and effect-size estimation. The synthetic pilot has not narrowed the human study to a single question; it has made the human study worth running by removing the upstream failures that would have made it uninterpretable.
Step 7. Feed discrepancies back into the lifecycle. Where synthetic and human results agree, the agreement is calibration data for which synthetic audits predicted human findings. Where they disagree, the disagreement is calibration data for where the synthetic-only pipeline did not generalize. Neither agreement nor disagreement closes the loop by itself.
The lifecycle is not a checklist that produces validity at the end. It is a sequencing discipline that allocates evidence-gathering effort to the cheapest places it can produce informative results, while protecting the integrity of the expensive validation step.
The Laundering Risk
The central operational risk of synthetic-first evaluation is laundering: presenting methodology-pilot results as if they were validation results, or letting a synthetic pilot's apparent success substitute for the human study it was supposed to enable.
Laundering happens through specific mechanisms.
Post-hoc harness tuning without a change log. A team runs the synthetic pilot, sees an unfavorable result, revises the rubric, the judge prompt, the scenario set, or the analysis specification, reruns, and reports the second run. Without a change log and a preregistration, the second run's apparent success is downstream of unregistered design choices, and the later human study — if it happens at all — is no longer confirmatory.
Synthetic judge agreement as evidence. A team reports that three LLM judges agreed on a verdict and presents the agreement as evidence of validity. LLM-judge agreement is evidence about the judges; it is not evidence about humans. Judges trained on overlapping data with overlapping reward models can agree confidently and wrongly.
Rubric optimization in the absence of human anchors. A team iterates the rubric against synthetic responses until the rubric produces stable scores. Without human anchor examples, the resulting rubric measures what LLMs reward, not what humans recognize. The rubric's apparent stability is a feature of the LLM-LLM loop.
Claim ladders that jump categories. A team reports a methodology pilot, the methodology pilot finds favorable signals, the report's framing drifts from "the harness now measures cleanly" to "the intervention works for users." The drift can happen in a single paragraph: "the synthetic pilot found a 7-point advantage" followed by "we recommend the conditioning be deployed." Each sentence is defensible in isolation. The combination is laundering.
Endless synthetic iteration as a delay tactic. A team uses each new synthetic finding as a reason to revise the harness one more time before recruiting humans. Each revision is individually defensible; the cumulative effect is that the human study never happens, and the team becomes increasingly committed to conclusions that have never been tested against the target population.
The mitigations are structural. Preregister which decisions can be made on the basis of synthetic findings and which must be held for human confirmation. Maintain a change log of every harness revision and the synthetic finding that motivated it. Set a maximum number of synthetic redesign cycles and a mandatory human shadow gate at the end. Report synthetic judge agreement as judge behavior, never as evidence about humans. Write the claim-boundary vocabulary into the report templates so that the discipline is enforced at the editing layer rather than the integrity layer.
The strongest mitigation, and the hardest one culturally, is to publish that the synthetic pilot found measurement fragility rather than to hide the fragility behind a polished benchmark. PsycheEval's v0.1 and v0.2 reports model this: the headline finding is what the methodology surfaced, not what the intervention appeared to do. Teams that want to do synthetic-first well copy this report shape. Teams that want to do synthetic-final cover it with a different headline.
The Bridge Sample
The decisive next experiment for any synthetic-first program is not a large human RCT. It is a preregistered bridge sample: a small set of high-leverage pairs from the synthetic pilot, judged by both the LLM-judge panel and a small set of blinded human raters, with the directional-agreement and rank-correlation statistics declared in advance.
A reasonable bridge sample design is 50–100 pairs selected before seeing human results, with the selection criteria covering the cases the synthetic pilot already informed: pairs where the synthetic judges agreed strongly, pairs where channels disagreed, pairs where length was a plausible confound, pairs where red flags drove the verdict, and a random sample for unstratified comparison. The primary endpoint is directional agreement between LLM-judge verdicts and human-rater verdicts; the secondary endpoints include rank correlation, channel-by-channel agreement, and subgroup analyses for the rubric dimensions that the synthetic pilot found separable.
The bridge sample's role in the lifecycle is to settle one specific question: does the synthetic-tuned harness produce verdicts that track human verdicts? If directional agreement is high and rank correlation is reasonable, the harness has earned the right to estimate effect sizes in a later shadow-mode or RCT study. If directional agreement is low or rank correlation is poor, the harness is useful for debugging the intervention but not for measuring it; the team's claims have to retreat to what the bridge sample supports.
The bridge sample is not a substitute for the validation study. It is the smallest empirical test that distinguishes a synthetic-first program from a synthetic-decoration program. A team that runs the bridge sample and reports its results — including unfavorable results — is doing the discipline. A team that defers the bridge sample indefinitely while the synthetic pilots accumulate is, increasingly, doing something else.
When Synthetic-First Doesn't Generalize
The article would be incomplete without naming the conditions under which synthetic-first is the wrong default.
Embodied or stakes-sensitive constructs. When the construct of interest depends on lived experience, group identity, embodiment, clinical stakes, or long-term social context in ways that simulation cannot capture even in principle, synthetic-first is weak. A synthetic persona cannot experience grief, financial precarity, chronic illness, social exclusion, or moral injury in any sense that licenses synthetic-pilot inference about how a real person in that condition responds to an AI system. The methodology can still surface harness bugs, but the validation question dominates the program, and resources should shift earlier to human work.
Generator-intervention coupling. When the intervention under test is itself a language model from the same family as the generator or the judge, the synthetic pilot risks measuring within-family preferences rather than properties of the intervention. The mitigation is cross-family generators and judges, which reduces the problem without eliminating it. In high-stakes settings, the right move is to treat the synthetic pilot as harness audit only and to pull validation forward.
Inability to freeze the harness. When the team cannot commit to a frozen protocol before human data collection, synthetic-first becomes a confirmation-bias amplifier. If every synthetic finding can revise the rubric, the judge prompt, or the scenario set indefinitely, then the eventual "validation" study is downstream of so many unregistered design choices that it is not confirmatory in any useful sense. Synthetic-first requires preregistration discipline as a precondition; teams that cannot do preregistration should not claim to be doing synthetic-first.
Population coverage outside the generator's competence. When the target population includes subgroups, languages, dialects, expertise levels, or cultural contexts that the generator represents poorly, the synthetic personas underweight or mis-represent the exact cases that drive deployment risk. The synthetic pilot will look clean and the deployed system will fail on the missing subgroups. Mitigation requires explicit coverage analysis against documented population statistics, plus targeted human work for the subgroups the generator misses.
Domain rarity. When the domain's failure modes are rare, idiosyncratic, or only legible to domain experts, no synthetic pilot will surface them at the rate needed to constrain the harness. Synthetic-first remains useful for the structural audits — judge bias, rubric separability, pipeline bugs — but the domain-specific failure mode discovery has to come from expert human review, not from scaled synthetic generation.
The honest framing is that synthetic-first is the right default for evaluation programs whose constructs are partially simulable, whose populations are within the generator's representational range, whose harnesses can be frozen before validation, and whose teams can resist the temptation to launder methodology results into validation claims. For programs that fail any of those conditions, the lifecycle should be re-weighted toward earlier human work, with the synthetic phase reduced to a structural audit of the harness rather than a substantive exploration of the intervention.
Open Questions
Three questions remain genuinely open in the synthetic-first literature and practice.
Predictive validity across programs. No published evidence base yet supports a general claim that synthetic-pilot findings predict human-study findings across multiple evaluation programs, domains, or populations. PsycheEval's bridge sample, when it is run, will be one data point. A defensible claim about synthetic-first as a general methodology requires many such data points, paired across programs that varied in synthetic design, target population, intervention type, and outcome channel. Until those data exist, claims about synthetic-first's predictive validity are theoretical, and individual programs should treat the bridge sample as a non-optional empirical check rather than as a formality.
The right interface between synthetic and human studies across the lifecycle. It is clear that synthetic pilots should precede human studies. It is less clear how synthetic and human work should interface after the first human study. Possibilities include: synthetic continuous monitoring of the deployed harness, with periodic human anchor recalibration; synthetic pre-screening of treatment arms before human RCTs of the most promising arms; synthetic adversarial probing of deployed systems with human follow-up on found failures; or synthetic methods work in parallel with permanent human in-the-loop evaluation. The right interface probably depends on the program; the field does not yet have enough lifecycle data to recommend defaults.
Whether synthetic-first delays or accelerates real validation. The optimistic reading is that synthetic-first concentrates harness-debugging cost in the cheap phase and shortens time-to-defensible-validation. The pessimistic reading is that synthetic-first becomes an indefinite delay, with each cycle of synthetic auditing producing new harness revisions that justify postponing human work. The empirical question is whether teams using synthetic-first reach human validation faster or slower than teams that skip the synthetic phase, controlling for program complexity. The honest answer is that the evidence base is small and the cultural pressure on programs to look methodologically mature can pull either way. The mitigation is the lifecycle gates: maximum redesign cycles, mandatory human handoff dates, and reports that distinguish "the harness is ready for human validation" from "the harness has been audited again."
The best available position on synthetic-first evaluation is therefore that it is a defensible default for a specific class of evaluation programs, with specific operational disciplines, specific failure modes, and specific empirical limits. It is not a paradigm shift. It is the simulation-before-experiment tradition adapted to a measurement context where the simulator and the target are uncomfortably close in surface form, and where the discipline of keeping the two separate is the difference between methodology engineering and methodology-flavored confirmation bias.
Companion entries
Core theory: - Construct Validity in AI Evaluation - Latent Construct Validity in AI Evaluation - Ecological Validity for AI Systems - Algorithmic Fidelity (Argyle et al.) - Goodhart's Law in AI Systems - ADEMP Framework for Simulation Studies - Methodology Pilot vs Product Validation - Generative Social Science (Epstein)
Practice: - Eval-Driven Development - LLM-as-Judge: Mechanisms and Limits - Judge Halo and Dimension Separability - Position Bias in LLM Judges - AB/BA Counterbalancing in Pairwise Eval - Length Confounds in LLM Evaluation - Channel Disagreement in Multi-Channel Eval - Bridge Sample Designs for Synthetic-to-Human Calibration - Preregistration for AI Evaluation Programs - Held-Out Blind Sets in AI Evaluation - Shadow Mode Evaluation - Judge Calibration Against Human Labels - Statistical Significance for LLM Evaluation
Adjacent constructs: - Red-Teaming as Stress Test, Not Validation - Capability Evaluation - Safety Evaluation - Production LLM Evals - Apparent Personality from Text - Caricature Under Persona Conditioning - Human-in-the-Loop Evaluation - Simulated Participants in Education Research
Counterarguments: - The Self-Confirmation Critique of Synthetic-First - LLM-Simulated Survey Respondents: Limits and Failures - Causal Inference Limits for LLM-Simulated Subjects - Synthetic-Final as a Failure Mode - Endless Methodology Iteration as Validation Delay
Case studies: - PsycheEval v0.1 Methods Pilot - PsycheEval v0.2 Hard Pilot and Counterbalancing Findings - PsycheEval v0.3 Planned Human-Rater Bridge