External validity asks where an evaluation result travels. In AI, the result usually travels much further than its evidence supports — from a fixed task distribution under a fixed elicitation regime to broad claims about capability, productivity, and deployment readiness. This article treats external validity as a structured transport claim, catalogs the specific failure modes that recur in AI evaluation, surveys the methodology responses and their second-order risks, and ends on the unresolved question of whether external validity is achievable for general-capability AI or whether the deployment-distribution problem is intrinsic.
Coverage note: verified through May 10, 2026.
1. What external validity is, in measurement-theory terms
In the experimental-validity framework of Donald Campbell, Thomas Cook, and William Shadish, validity is decomposed into four types: statistical conclusion validity, internal validity, construct validity, and external validity. External validity is defined as the validity of inferences about whether a causal relationship holds over variation in persons, settings, treatment variables, and measurement variables — that is, generalization beyond the conditions actually sampled in the study (Shadish, Cook & Campbell 2002). It is the validity of transport.
In Samuel Messick's unified validity theory, the distinction softens but does not disappear. Messick treats validity as a single integrated argument about the warranted interpretation and use of measurements, with generalizability and external relations as evidence inside that argument rather than as a separate validity type (Messick 1989). The AERA/APA/NCME Standards for Educational and Psychological Testing follow Messick: validation is an evidentiary program; external validity language is one strand inside it, concerned with whether score interpretations hold across the populations, contexts, and uses claimed (AERA, APA & NCME 2014).
Two implications carry into AI evaluation. First, external validity is claim-relative: it does not attach to a benchmark, it attaches to an inference being drawn from a benchmark for a specific use. The same SWE-Bench result is externally valid for the inference this model variant has not regressed at producing patches to mature Python repositories under standardized harness conditions and externally invalid for the inference this model will improve our team's coding throughput. Second, external validity is separable from but coupled to Construct Validity: a benchmark can measure the right construct on the wrong population, the wrong construct on the right population, or fail at both. The companion entry on construct validity treats the what is measured question; this entry treats the where does it travel question. The two failure types usually appear together in critique because the inferential moves they prohibit usually appear together in practice.
A useful taxonomy distinguishes external validity from adjacent generalization concepts:
| Concept | Asks |
|---|---|
| External validity | Does the inference hold across populations, settings, treatments, and outcomes beyond those sampled? |
| Ecological validity | Does the measurement setting resemble the deployment setting in materials, demands, and constraints? |
| Population validity | Does the inference hold for the population of interest given the sampled population? |
| Temporal validity | Does the inference hold across the time period of interest? |
| Criterion validity | Does the score predict an independent outcome that matters? |
| Generalizability theory | A formal decomposition of variance over facets (raters, items, occasions, etc.) into the variance an inference depends on. |
Ecological validity is sometimes used as a synonym for external validity in casual AI-evaluation discourse. It is a narrower concept: it asks whether the measurement environment is plausibly real, not whether inferences from it transport. A high-fidelity simulation of a coding workflow can be ecologically valid (it looks like the real thing) and externally invalid (the inference still does not generalize because the user population, task distribution, or deployment incentives were not sampled). Conversely, an abstract benchmark with little ecological resemblance to deployment can support a narrow externally valid inference if the inference is appropriately scoped. The two are correlated but not identical, and the AI literature loses precision when it conflates them.
2. Why AI evaluation makes external validity unusually hard
The Campbell/Cook/Shadish framework was developed for randomized experiments on bounded populations — clinical trials, education programs, social interventions. Even there, external-validity claims are demanding: real target populations are hard to sample, real outcomes are mediated by socio-technical context, and effect sizes vary across subgroups in ways that simple aggregate estimates obscure. AI evaluation adds at least four further difficulties on top of the standard ones.
The deployment distribution is open-ended. A clinical trial of a drug for type 2 diabetes has a target population that is conceptually bounded: adults meeting diagnostic criteria for type 2 diabetes within some health-system context. The eligibility criteria can be specified, even if sampling is hard. A general-purpose language model has no such bound. The "target population" of prompts is the union of everything anyone might type into the model, mediated by every product surface it gets integrated into, refracted through every workflow that organizations build around it. There is no eligibility criterion. The target population of tasks, in particular, is not a sample from a fixed distribution; it is a sample from a distribution that the model partly creates by enabling new workflows.
The deployment distribution is endogenous to the model. Users adapt to model affordances. Organizations redesign their workflows around what models can do. Developers build scaffolds that change the tasks that reach the model. Adversaries probe edges, and the model provider patches behavior in response. Public knowledge of benchmarks changes both training (through contamination and selection) and prompting (through community-shared techniques). The independence assumption that underlies straightforward inference — that the population being measured does not depend on the measuring instrument — is partly violated for AI in ways it is not for most other measurement domains. This is one of the senses in which "general-capability" evaluation is unusually fragile: the act of releasing the model changes the population the model is being claimed to be capable across.
Performance is socio-technical, not model-intrinsic. Real-world outcomes for AI systems are produced jointly by the model, the prompt, the decoding parameters, the tool stack, the retrieval layer, the agent scaffold, the user, the user's organization, the review and correction process, and the time available to complete the task. Attributing observed performance differences to "the model" — let alone treating them as evidence about model capability that transports across deployments — requires assumptions about which parts of the stack are held constant. Those assumptions are rarely defensible at the granularity of "capability" claims, and the literature has begun to take this seriously: the AI Functionality critique of Raji and colleagues argues that "passing research evaluations is not the same as functioning in deployment" because deployment functionality requires the full socio-technical context to work, not just the model component (Raji et al. 2022).
Interaction protocol matters at first order. A model that answers a single fully specified prompt well may fail when it must clarify, maintain context, recover from earlier errors, decide when to ask for help, or coordinate tool calls. A 2025 study by Laban, Hayashi, Zhou, and Neville sharpens this with explicit measurement: across 15 LLMs and six generation tasks, the same models that scored well on fully specified single-turn prompts dropped by an average of 39 percentage points when identical task content was delivered as underspecified multi-turn conversations, with a smaller (-2pp) drop in average capability but a much larger (-50%) drop in average reliability across turns (Laban et al. 2025). The interaction regime is not a context detail; it is a first-order variable that determines whether an external-validity claim about a model is meaningful.
These four difficulties do not make external-validity reasoning impossible. They make it locally tractable and globally fragile. The natural form of an external-validity argument for an AI system is bounded — for this model variant, on this task family, for this user population, under this interaction protocol, within this time window — and the failure mode that dominates current discourse is the move from a bounded inference to an unbounded one.
3. Five axes of generalization
A useful reusable mental model is to decompose every external-validity claim into the axes it is implicitly making. Five axes cover most AI cases.
| Axis | What the inference is generalizing over | Typical failure |
|---|---|---|
| Population | Users, developers, organizations, languages, cultures, domains, query distributions, even other models that the system interacts with | Expert-lab or crowd-worker results are transported to "users" without sampling them; English-only results are transported to multilingual products; tasks authored by ML researchers are transported to tasks authored by end users |
| Task | Task family, difficulty distribution, length, structure, ambiguity, and the share of tasks that are well-specified versus underspecified | Contest-style or puzzle tasks are transported to real production work; closed-book tasks are transported to deployments with retrieval; well-specified prompts are transported to underspecified ones |
| Setting | Tools, retrieval, latency, cost constraints, governance rules, incentives, review processes, organizational context | Clean lab conditions are transported to real socio-technical environments; bench-style "model alone" measurements are transported to agent stacks; unmonitored eval conditions are transported to environments with logging, oversight, and operational constraints |
| Interaction | Single-turn vs. multi-turn, tool-use scope, agentic vs. supervised, monitored vs. unmonitored, collaborative vs. solo | Single-shot benchmark performance is transported to long-horizon agent behavior; full-specification prompts are transported to underspecified ones |
| Time | Model snapshot, benchmark contamination state, user adaptation stage, threat environment, regulatory environment | Pre-release benchmark results are transported across model updates; post-release results are transported across user adaptation; static benchmarks are transported across an evolving deployment landscape |
A clean external-validity claim names a slice along each axis. This score is evidence that model variant V, given access to tool T and retrieval R, produces correct patches for issues drawn from the Python-language subset of the SWE-Bench-Verified distribution under single-attempt scoring as of May 2026 is a well-specified inference. Model V is better at coding is not. The implicit-axis problem is the source of most overreach: the inference is made along several axes simultaneously without the evidence to support transport on any of them.
The axes also clarify what a benchmark is good for. A static public benchmark is reasonably well-suited for within-family comparison — does this model variant score higher than the previous variant on the same task distribution under the same elicitation protocol? — and ill-suited for population, setting, and interaction generalization. This distinction underlies the Two-Track Account of Benchmarks: one track for engineering instruments, where validity is local and inferences are narrow; another track for capability claims, where validity is global and inferences are broad. Many of the most contentious benchmark debates concern benchmarks that began life on the engineering track and migrated, through external citation, onto the capability track.
4. Specific failure modes in AI evaluation
The recurring external-validity failures in AI evaluation can be organized by the axis along which the inference fails. The pattern across the literature is consistent: each failure type is well-documented as a phenomenon and badly documented as a quantified transport relationship. Knowing that lab tasks do not always transport to deployment is established; knowing the magnitude and direction of the transport gap for any specific class of inference is mostly not established.
4.1 Benchmark-to-deployment mismatch
A benchmark is a sample from some implicit task distribution. Deployment is a sample from a different task distribution. External validity from benchmark to deployment requires either that the two distributions are close enough that inferences transport, or that the inference is narrow enough that the gap does not matter. Public benchmarks tend to violate the first condition systematically: they are typically authored by researchers, scored automatically, designed to be unambiguous, drawn from text on the open web, biased toward English, and toward task types that are convenient to evaluate. Deployment tasks are authored by end users, scored by their downstream effects, often ambiguous, drawn from private context, multilingual, and biased toward task types that the product surfaces well.
John Miller's 2022 UC Berkeley thesis on distribution shift quantifies the underlying problem. Across many image-classification and NLP settings, Miller documents that even small distribution shifts produce large performance drops, that benchmark performance ranks are not robust to plausible shifts, and that the test-set assumption that the evaluation sample is exchangeable with the deployment sample is routinely violated (Miller 2022). Liao, Taori, Schmidt, and Raji's earlier review of 107 ML survey papers catalogued external-validity failures across application areas, finding that benchmark-instantiated learning problems rarely capture the broader problems they purport to instantiate (Liao et al. 2021). The empirical record across these reviews supports a strong default: benchmark-to-deployment transport is a claim that needs evidence, not the null hypothesis to be assumed.
The most useful response is scope discipline. A benchmark result licenses an inference within its evidence envelope. The envelope is set by the benchmark's actual sampling frame, not by its name. SWE-Bench is evidence about model behavior on issues drawn from popular Python repositories under specific elicitation conditions; it is weak evidence about coding ability in general, and the move from the former to the latter is the failure to police.
4.2 Lab-task non-transfer to real coding work
Coding is the cleanest case study for external-validity failures because the lab-task evidence and the field evidence both exist and disagree in informative ways.
Peng, Kalliamvakou, Cihon, and Demirer's 2023 study of GitHub Copilot on a constrained task — implementing an HTTP server in JavaScript — found that Copilot users completed the task 55.8% faster than the control group, with a 95% confidence interval of 21% to 89% (Peng et al. 2023). The task is well-defined, scoped, and roughly representative of a narrow class of self-contained coding problems. Within that envelope, the result is robust.
Cui, Demirer, Jaffe, Musolff, Peng, and Salz's 2026 analysis of three enterprise field RCTs (Microsoft, Accenture, and an anonymous Fortune 100 firm), covering 4,867 developers, found an estimated 26.08% increase in completed tasks among developers given Copilot access (Cui et al. 2026). The result is field rather than lab, sampled from actual work in three large engineering organizations, and consistent in direction across all three sites. Less experienced developers benefited more in this dataset than more experienced developers, which is itself an externally interesting result about who the tool helps.
METR's "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" reports a result in the opposite direction. METR's RCT used 16 experienced open-source developers working on 246 real tasks drawn from mature open-source repositories with which the developers were familiar. AI use was randomized at the task level. Developers' pre-task forecasts predicted a 24% speedup; their post-task self-reports estimated a 20% speedup; the measured effect was a 19% slowdown (Becker et al. 2025). The headline result is striking, but the more important contribution is structural: the paper documents how lab-style coding benchmarks, simple output metrics, and self-reports can move sharply in the opposite direction from the actual completion-time effect on real maintenance tasks in familiar repositories.
These results do not contradict each other. They are external-validity evidence from different points in the population × task × setting × interaction × time space. The Peng result transports to constrained, self-contained coding tasks performed by their study population at the time of measurement. The Cui results transport to enterprise developer populations on production work as of late-2025/2026 tooling. The METR result transports to experienced open-source developers on mature, familiar repositories as of early-2025 tooling. None of them transport, by themselves, to the unqualified claim AI improves coding productivity. The disagreement among the studies is the substance of the external-validity question for coding productivity, not a bug in any single study.
METR's February 2026 update is itself an external-validity lesson. The team reports that follow-on measurement on the same population became progressively harder to interpret because developers increasingly refused no-AI conditions, selection effects accumulated, and the task distribution shifted as workflows reorganized around AI assistance (METR 2026). The deployment distribution was changing under the measurement. This is precisely the endogenous-distribution problem at field-experiment scale: even a well-designed RCT in a stable population can lose external validity as the population adapts to the intervention it is studying.
The intellectually honest summary is that AI coding-assistance effects are heterogeneous across population, task, and time, and that any specific transport claim is an inference that must be argued, not asserted. Generic claims of the form AI makes developers N% faster are external-validity violations regardless of the sign of N.
4.3 Single-turn evidence transported to multi-turn behavior
The interaction-axis failure has become measurable. Laban, Hayashi, Zhou, and Neville's 2025 paper, "LLMs Get Lost in Multi-Turn Conversation," tests fifteen LLMs on six generation tasks under two conditions: fully specified (all task content delivered in a single turn) and sharded (the same task content split across multiple turns of an underspecified conversation). The sharded condition is constructed to preserve the construct — same information, same answer would be correct — while changing the interaction regime. Across the panel, the sharded condition reduced performance by an average of 39 percentage points, with the gap dominated by an increase in reliability variance rather than a drop in peak capability: models occasionally still answered correctly, but their best-case-to-worst-case spread widened sharply (Laban et al. 2025).
The methodological lesson is that interaction-regime substitution is a large effect, often comparable to the gap between model generations on the same benchmark. A claim of the form Model V has reasoning ability R that was measured on a single-turn benchmark cannot be transported to multi-turn deployment without separate evidence. The reasoning task may be the same and the model may be the same, but the interaction regime is not, and the regime change carries first-order performance consequences.
A weaker but related pattern is the gap between zero-shot, no-tools, no-memory benchmark conditions and agentic deployment with retrieval, browsing, code execution, and external state. Agent benchmarks themselves often suffer the inverse failure: sandboxed agent evaluations license claims about autonomy that have not been validated under deployment-realistic governance, cost, latency, recovery, and adversarial conditions.
4.4 Contamination, saturation, and benchmark aging
External validity along the time axis fails for reasons that include but go beyond Benchmark Contamination. The contamination mechanism is straightforward in one direction: if a benchmark's items have leaked into pretraining or post-training pressure, scores no longer indicate generalization to unseen items in the construct's domain — they indicate something closer to recall on items the model has effectively memorized. Recent contamination-resistant benchmark designs such as LiveBench and AntiLeakBench (treated in detail in Construct Validity's contamination section) are attempts to restore the held-out assumption that external validity along the time axis requires.
The harder problem is saturation combined with selection. As benchmarks become salient, the population of models being released is selected on benchmark performance, which means the model population is no longer independent of the benchmark population. Aggregate leaderboard improvement no longer cleanly indicates improvement in the underlying construct; it indicates improvement as the construct is operationalized by the benchmark, conditional on the selection pressure the benchmark has exerted. This is an external-validity failure even when no contamination has occurred: the inference from leaderboard improvement to capability improvement loses warrant as the field optimizes for the benchmark.
A third time-axis failure is workflow drift. Even when the benchmark is stable, the deployment landscape it claims to transport to is not. Tool stacks change, prompting techniques become standard, agent scaffolds become common, and product surfaces change which tasks reach the model. A benchmark result that was a defensible proxy for deployment behavior in early 2024 may be a weaker proxy in late 2025 because the deployment surface has changed underneath it.
4.5 Cultural, linguistic, and domain narrowness
External validity along the population axis fails most visibly when benchmarks are constructed in English by US-based research groups, scored by US-based annotators, drawn from US-skewed text on the open web, and then used to support claims about model behavior in markets that look nothing like that. The corrective is straightforward — multilingual evaluation, cross-cultural item construction, domain-specific benchmarks — but the corrective is still missing for many high-stakes capability claims. The systematic review by Bean and colleagues of 445 LLM benchmarks documents the structural sparsity of population-axis evidence reporting at corpus scale (Bean et al. 2025).
This failure mode interacts with construct validity. A benchmark that measures the right construct in English on US-authored items can be construct-valid for English-language US deployment and externally invalid for deployment in other languages and contexts. Naming the failure as external rather than construct is the right call when the construct itself transports in principle but the operation has only sampled one slice of its domain.
5. Methodology responses
The methodology responses to external-validity failures are well-known in measurement theory. The AI-evaluation-specific question is how each response is implemented in practice, what its second-order risks are, and which inferences it can and cannot license. None of the responses are neutral fixes; each has its own failure modes and governance burden.
5.1 Target-population audit
A target-population audit is the entry ticket for any external-validity claim. The procedure is to specify, before measurement, the target population the inference is supposed to cover: who the users are, what tasks they perform, in what setting, under what interaction protocol, during what time window, with what tool stack, under what success metric. The specification is then matched against the eval sample to identify gaps.
The audit is straightforward to describe and hard to execute. The hard parts are usually that the target population is not knowable in advance (the model has not been deployed yet), or that it changes after deployment (endogeneity), or that it spans more variation than any feasible sample can cover (general-purpose deployment). The honest output of a target-population audit is often the recognition that the broad claim being made is unsupportable and must be narrowed before it can be validated.
A target-population audit is decisive for bounded deployment claims of the form this support bot for this product, for these users, during this release window and indecisive for composite construct claims like this model is good at coding whose deployment populations are not merely unsampled but not fixed. The discipline the audit imposes is to narrow claims to a scope where evidence can be brought to bear. It is not, by itself, a method for generating evidence.
5.2 Deployment-distribution sampling
Deployment-distribution sampling draws eval items from the actual distribution of usage, typically through telemetry, trace logging, or post-hoc sampling of production conversations. The strongest form is stratified sampling by user segment, task type, and outcome class, with explicit attention to rare-but-important tasks.
The second-order risks are substantial. Existing telemetry overrepresents existing users and tasks; deployment-distribution sampling can preserve the current user base's blind spots and exclude users and tasks that should be served but currently are not. Sampling by observed frequency improves average-case prediction while underweighting the tail-heavy distribution of high-impact failures. Privacy and consent constraints limit which traces can be inspected, and the subset that survives consent filtering is itself a non-random sample. Telemetry gaps — operations that are not logged, conversations that fail before reaching the model, errors that are silently retried — create systematic exclusion from the sample.
Used carefully, deployment-distribution sampling is the most direct evidence available about the distribution that real deployment generates. Used uncarefully, it laundrers the existing usage distribution into "ground truth," which is a different failure mode from benchmark over-reach but a comparable one.
5.3 In-the-wild evaluation
In-the-wild evals run actual deployments under instrumented conditions — A/B tests, shadow-mode comparisons, staged rollouts with monitored outcomes. They are typically the strongest evidence about deployment behavior, because the inference is about deployment behavior and the measurement is taken during deployment.
The risks come in three layers. The first is selection: users who opt in to instrumented conditions, or who happen to be in the rollout cohort, are not random. The second is observer effect: users who know they are being evaluated change their behavior, and developers who know their tasks are being measured do too. The METR 2026 update is a clean example of an observer-effect failure mode: developers refused the no-AI condition once the productivity stakes became salient, and the eligible task distribution narrowed in ways that confounded the measurement (METR 2026). The third is governance: in-the-wild evaluation requires consent infrastructure, privacy protection, incident protocols, and protections against unjustified harm — all of which are easier to specify than to implement well, and all of which constrain what can be measured.
In-the-wild evidence is also vulnerable to the same socio-technical confounding as deployment-distribution sampling. A change in productivity, error rate, or user satisfaction during a rollout reflects the joint effect of model, prompt, scaffold, user adaptation, organizational learning, and the rollout itself. Attributing the effect to "the model" requires the kind of design discipline that randomized field experiments demand and that A/B-test infrastructure often does not provide.
5.4 Holdout populations
Holdout populations — groups of users, organizations, or tasks deliberately reserved from the training and post-training pressure that the model has been exposed to — are the closest AI-evaluation analogue to clinical-trial holdouts. They support inferences about transport across populations that the model has not been adapted to, and they protect against the saturation/selection failure that public benchmarks suffer from.
The risks are governance-heavy. Holdouts require access control, audit trails, and protection from leakage; once a hidden benchmark becomes influential, it attracts informal optimization pressure, and the population it samples can stop being representative as the field absorbs information about its contents. Holdouts also require ongoing maintenance: a hidden population frozen at a point in time slowly drifts away from the present-day deployment landscape, becoming a measure of "how the model performs on slightly stale tasks" rather than "how the model performs on current ones."
The combination of holdout populations with periodic refresh is the most defensible operational form, but it is expensive and exposes the host organization to disclosure and security risk. For high-stakes capability claims, the cost is usually worth paying. For everyday model selection, lighter-weight evidence is often sufficient.
5.5 Longitudinal monitoring
Longitudinal monitoring runs the eval continuously over time, with results recorded against model version, scaffold version, tool stack, and population sample. The value is twofold: it makes time-axis external validity tractable by measuring the transport relationship directly, and it surfaces drift before it becomes a deployment failure.
The risk is that what gets monitored is what is easy to monitor. Time-on-task is easy; rework, security implications, downstream coordination cost, and long-term maintainability are hard. Reducing "productivity" or "quality" to a single time-series metric is an external-validity failure dressed up as rigor. The longitudinal evidence supports an inference only if the outcome metrics it tracks actually instantiate the construct the inference is about.
5.6 Validity envelopes and the triangulation trap
A practical synthesis is to require every eval result to publish a validity envelope: a structured specification of the population, task family, setting, interaction protocol, time window, exclusions, and supporting evidence that the claim covers. The envelope is a substitute neither for evidence nor for argument; it is an artifact that forces both to be made explicit. When followed, it converts the question from is this benchmark externally valid? (unanswerable in the abstract) to does this evidence support the inference inside the envelope? (answerable in detail).
The trap is that validity envelopes can become boilerplate. A sentence reading valid for English-language users on standardized tasks in supervised conditions attached to every result, with no falsification signals and no explicit exclusions, is decorative rather than load-bearing. The envelope is doing real work only when it produces negative space: this evidence does not support claim X, this evidence does not transport to user population Y, the inference will need to be re-validated when interaction regime Z changes.
A second trap concerns triangulation. The intuition that combining multiple imperfect evidence sources produces stronger inferences is correct when the error modes of the sources are independent. It is false when they share a blind spot. If benchmarks, in-the-wild traces, holdout populations, and longitudinal monitoring all draw from the same user base, the same task distribution, the same product surface, and the same instrumented setting, their agreement does not increase confidence in transport beyond that setting — it only increases confidence in transport within it. Triangulation is a real validity strategy when the sources have distinct sampling frames and distinct failure modes. When they do not, it functions as a way to launder shared bias into apparent robustness.
The defensible operational stance is that methodology responses narrow claims and audit evidence; they do not, on their own, convert a broad benchmark result into a broad transport claim. The hard work is the narrowing, and it is best done before the result is published, not after the result is being cited.
6. Claim type to minimum evidence
The validity-argument frame becomes operational when it specifies, for each claim type, the minimum external-validity evidence the claim would need to be defensible. The table below is a starting point, not a closed catalogue.
| Claim type | Minimum external-validity evidence |
|---|---|
| Model A scores higher than Model B on benchmark X | Benchmark protocol; reliability under repeated administration; rank stability under construct-preserving prompt perturbation. No transport claim is being made. |
| Model A is better at task family T than Model B | Above, plus convergent evidence across independently designed measures of T, and disaggregated performance across the population of items within T. |
| Model A will perform better on our internal workflow W | Trace-sampled evaluation of A on tasks drawn from W's actual distribution, under the actual tool stack and interaction protocol of W, with a comparable B-condition. |
| Model A improves developer productivity in organization O | Field experiment within O on real backlog tasks with crossover or randomized assignment; outcome metrics that include completion time, rework, defect leakage, review burden, and at least one quality measure; pre-registered analysis plan. |
| Model A improves developer productivity generally | Multiple field experiments across independent organizations, task families, and developer experience levels; preregistered claims; published negative as well as positive results; explicit transport argument for the population the "generally" claim is supposed to cover. |
| Model A is safe for biomedical advice | Adversarial probing along documented threat models; in-the-wild evaluation in supervised clinical settings; incident-reporting infrastructure; population coverage across patient populations, conditions, and decision contexts the claim is supposed to cover. Single-benchmark evidence is structurally inadequate. |
| Model A is more capable than Model B | A claim of this form, without specification of the population/task/setting/interaction/time axes, is not externally valid in any form. The honest response is to ask what specific inference is being made. |
The bottom row matters. The most common external-validity failure is not the construction of bad evidence for a broad capability claim; it is the assertion of a broad capability claim that has no specifiable evidence requirements because its target of inference is not specified. A "capability" claim without an attached population, task, setting, and interaction regime is not a claim that can be externally valid or invalid. It is a claim that has not yet been made.
7. The productivity literature as a sustained external-validity test
The AI coding-productivity literature has accumulated enough evidence to function as the field's most legible external-validity test case. Three results define the current shape of the debate.
Peng, Kalliamvakou, Cihon, and Demirer's 2023 Copilot RCT is the high-end of the lab-task evidence: on a constrained, well-specified JavaScript task, treatment-group developers completed the task 55.8% faster than the control group (Peng et al. 2023). The result is internally valid and externally valid within its envelope: it supports inferences about Copilot's effect on developers performing self-contained coding tasks of the kind sampled in the study, under the conditions the study controlled. It does not support the inference AI makes coding faster in general because the population, task, and setting axes are not jointly sampled to cover the "general" claim.
Cui, Demirer, Jaffe, Musolff, Peng, and Salz's 2026 enterprise field analysis is the high-end of the in-the-wild evidence: across three field RCTs with 4,867 developers at Microsoft, Accenture, and an anonymous Fortune 100 firm, Copilot access produced an estimated 26.08% increase in completed tasks (Cui et al. 2026). The result transports to enterprise software engineering populations on production work as of the late-2025/early-2026 tooling generation. It also reports heterogeneous effects: less experienced developers benefited more than more experienced developers, and the gap was large enough to suggest a population-axis transport question (does the average effect mask diverging effects by experience strata?).
METR's "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" is the high-end of the high-context evidence. Sixteen experienced open-source developers worked on 246 real tasks drawn from mature open-source repositories with which the developers had deep familiarity. AI access was randomized at the task level. The developers' pre-task forecasts predicted speedup; their post-task self-reports estimated speedup; the measured completion-time effect was a 19% slowdown (Becker et al. 2025). The paper's contribution to external-validity discourse is not the headline number — it is the demonstration that lab-task results, forecasts, and self-reports can move sharply in the opposite direction from the field result on a different point in the population × task × setting space.
The intellectually honest reading is that these results are all externally valid for the inferences they license inside their envelopes and all externally invalid for the unqualified claim. AI coding assistance can improve productivity 55.8% on constrained tasks for the Peng study population, increase task completion 26.08% in the Cui enterprise populations, and slow experienced open-source developers by 19% on mature repositories, simultaneously, without contradiction. The heterogeneity is the topic. A claim of the form AI makes coding faster that cites the Peng or Cui result without sampling the population the claim is being made over is an external-validity failure regardless of the direction of the cited effect. So is a claim of the form AI does not help coding that cites the METR result without acknowledging that the experienced-OSS-developer-on-familiar-repo population is itself a slice.
METR's February 2026 update is the field's most explicit lesson in endogenous-distribution external-validity failure. The follow-on measurement program found that developers increasingly refused the no-AI condition when the productivity stakes became visible; the population that remained available for measurement was a self-selected subset; the task distribution that reached the no-AI condition shifted as developers triaged on perceived AI suitability; and the measurement program's ability to estimate the AI effect degraded as a result (METR 2026). This is a field-experiment-scale demonstration of the broader pattern: the deployment distribution changes under the measurement, and external-validity claims based on a snapshot of the distribution lose force as the snapshot ages.
The productivity literature is the cleanest cautionary example for the same reason it is the cleanest positive example: the evidence is heterogeneous, the heterogeneity is structured by external-validity axes, and the temptation to collapse the heterogeneity into a single headline number is the failure mode the article is about.
8. Is external validity achievable for general-capability AI?
The article's final question is whether external validity is achievable for general-capability AI, or whether the deployment-distribution problem is intrinsic to general-purpose systems.
The strong skeptical position holds that the question is mis-posed. External validity is a property of an inference about transport from a measured setting to a named target population. For general-capability AI, the target population is not a fixed distribution that can be sampled; it is the union of all tasks anyone might ever attempt with a general-purpose model, including tasks the model itself enables by being available. This is not a sampling problem that better instrumentation can solve. It is a definition problem: there is no fixed target to be valid for. On this reading, "external validity of a general-capability benchmark" is a category error in the same way as "external validity of a thermometer for measuring intelligence." The instrument is being asked to support an inference that has no specifiable population of inference.
The strong recoverist position holds that general capability decomposes into a structured collection of sub-capabilities — mathematical reasoning under symbolic constraints, scientific problem-solving with tool access, long-horizon task execution under realistic context, calibrated factual recall, instruction-following under specified ambiguity, multi-turn dialogue with recovery from misunderstanding, agentic task execution under cost and latency constraints — each of which can be operationalized, sampled, and validated under standard machinery. The aggregate construct then inherits the validity of its decomposition: a general-capability claim is well-formed if it is the conjunction of well-validated sub-capability claims with explicit transport arguments between them. On this reading, external validity for general-capability AI is achievable in principle, partly achievable in practice for some sub-capabilities, and currently unachieved at the aggregate level.
The intermediate position, which the literature most strongly supports, holds that external validity is achievable for bounded deployment claims and intrinsically constrained for broad general-capability claims as currently stated. A bounded claim — this model variant, for this user population, on this task family, under this interaction protocol, in this tool stack, during this time window — is the kind of claim that target-population audits, deployment-distribution sampling, in-the-wild evaluation, holdout populations, and longitudinal monitoring can support. A broad claim — this model has general reasoning ability — lacks the specification required to be externally valid in any form, regardless of how much evidence is brought to bear. The intermediate position is not a compromise; it is a description of where the inferential machinery actually works.
This has a corollary that is uncomfortable for benchmark-driven discourse. A leaderboard ranking by aggregate score is, at best, an internally consistent comparison instrument; it is not, by itself, evidence about transport. Treating aggregate scores as a proxy for general capability is the move that the external-validity argument cannot license. The corollary is not that benchmarks should be abandoned. It is that benchmark results should be evaluated against the validity envelope of the inference being drawn from them. For within-family regression testing, an aggregate score is often sufficient. For deployment decisions about new populations, new tools, new interaction regimes, or new time periods, aggregate scores require supplementing with the methodology responses described above, and the supplementing is part of the inference, not optional decoration around it.
The remaining open question is whether better instrumentation can collapse the gap between bounded validity and broad validity over time. Two trajectories are plausible. The optimistic trajectory is that as longitudinal multi-site evaluation infrastructure matures, the field will accumulate enough preregistered benchmark-to-deployment correlation evidence across users, tools, interaction regimes, and model generations to support broader transport claims with confidence intervals. The pessimistic trajectory is that the endogeneity of the deployment distribution — users adapting to models, organizations redesigning workflows around models, model providers patching against documented failures, public discourse changing prompting practice — keeps the distribution unstable faster than the evidence base can describe it. The two trajectories are not mutually exclusive. Better instrumentation almost certainly narrows the gap on slow-moving axes (population and task) while leaving it wide on fast-moving ones (interaction protocol, scaffold composition, time).
The intellectually honest position is to treat the global external-validity question as open and operationally useful. It is open because no current benchmark family has demonstrated stable, preregistered benchmark-to-deployment correlation across users, time, tools, and institutions. It is operationally useful because the practical work of evaluation does not need a global answer: it needs bounded claims with bounded validity envelopes, audited evidence for transport, explicit acknowledgment of where evidence is thin, and a discipline of revising claims as the deployment distribution changes. The article's central recommendation is that "external validity" be treated not as a property a benchmark can attain but as a transport claim that must be specified, defended, and bounded for every inference it is asked to support.
The article does not recommend benchmark cynicism. Benchmarks are useful instruments for specific inferences. It also does not recommend leaderboard realism. Aggregate scores are not evidence of general capability without the validity argument that bridges the gap. What it recommends is claim discipline: state the inference, specify the target of generalization, identify the evidence, name the exclusions, expect the deployment distribution to move, and rebuild the argument when it does.
Companion entries
Core theory: - Construct Validity - Measurement Theory - Internal Validity - Ecological Validity - Generalizability Theory - Causal Inference for AI Evaluation
Practice: - AI Evaluation - AI Capability Evaluation - Benchmark Contamination - Deployment-Distribution Sampling - In-the-Wild Evaluation - Holdout Populations - Longitudinal Evaluation - Validity Envelope - Claim-Centered Evaluation - Two-Track Account of Benchmarks - Agent Evaluation - Multi-Turn Evaluation
Productivity evidence: - AI Coding Productivity - METR Developer Productivity Study - GitHub Copilot RCTs - Enterprise AI Field Experiments
Counterarguments and open questions: - General Intelligence (AI) - Capability Decomposition - Endogenous Deployment Distributions - AI Functionality - Benchmark Saturation and Selection Pressure