Abstract. AI evaluation has moved from static public benchmarks to dynamic, adversarial, and increasingly private production eval suites because benchmark scores are proxy measurements, not direct measurements of useful or safe behavior. GLUE, SuperGLUE, and MMLU made progress legible; GPQA, ARC-AGI, SWE-bench, LiveCodeBench, and related successors tried to repair known failure modes; and current practice at leading labs now treats evals less like immutable leaderboards and more like tests for system behavior. The central problem is therefore a measurement problem AI Measurement Problem: contamination, saturation, construct-validity failures, and Goodhart pressure all make it easier to optimize the score than to preserve what the score was supposed to mean. (arXiv)
Coverage note: verified through April 5, 2026.
Why AI evaluation became a measurement problem
A benchmark is a shared, standardized test; an eval is the broader class of procedures that feed inputs to a model or system and score outputs or outcomes; and a metric is the scoring rule inside that procedure. OpenAI’s current guidance explicitly separates industry benchmarks such as MMLU from reusable numerical scores such as ROUGE or BERTScore and from application-specific tests that teams build for their own systems, while Anthropic’s agent-evals guidance frames an eval simply as a test: give an AI system an input, then apply grading logic to what it does. That distinction matters because arguments about “evals” often collapse three different things into one: public comparison instruments, private regression suites, and safety or deployment gates. AI Benchmarks Capability Evaluation Safety Evaluation (OpenAI Developers)
That collapse is the beginning of the measurement problem Construct Validity. Once a score matters economically or culturally, the field does not just optimize the model against it; it optimizes the data mixture, prompting strategy, inference scaffold, agent harness, grader, and even the presentation layer around it. Benchmark-validity reviews now describe this as a systematic problem rather than a rare pathology: benchmark outcomes are sensitive to the benchmark chosen, many benchmarks make stronger claims than their construct validity supports, and one-shot public testing is a poor fit for interactive, multimodal, tool-using systems. (arXiv)
The resulting question is not merely “can the model do the task?” but “what still follows from success on this instrument?” That is a harder question. It requires separating external comparability from internal decision-making, broad capability estimation from safety assessment, and model-only performance from end-to-end system behavior. Vendor documentation now reflects that shift directly: both OpenAI and Anthropic emphasize defining success criteria, building task-specific tests, and rerunning them continuously rather than treating public benchmarks as sufficient release criteria. Production LLM Evals Eval-Driven Development (OpenAI Developers)
The static-benchmark era: GLUE, SuperGLUE, MMLU
The modern benchmark era in NLP was organized by GLUE. GLUE bundled nine language-understanding tasks into a single benchmark with an online evaluation platform and a diagnostic suite, making it possible to compare models on a common multitask target instead of on isolated tasks with incompatible reporting norms. Historically, that mattered as much as the task design itself: GLUE gave the field a public measurement object and a leaderboard culture. GLUE Leaderboard Culture (arXiv)
SuperGLUE appeared because GLUE stopped differentiating quickly. Its authors explicitly introduced a harder benchmark after models surpassed human baselines on GLUE within roughly a year, which is the first clear instance of a pattern that now repeats across the field: once a benchmark becomes central, progress pressure rapidly converts it from frontier test to maintenance problem. SuperGLUE Benchmark Saturation (arXiv)
MMLU then widened the notion of what a benchmark should cover. Instead of focusing on narrow NLU tasks, it sampled 57 academic and professional subjects, positioning benchmarking as a proxy for broad knowledge and reasoning. That move was historically important: MMLU became a lingua franca for “general capability” claims in large language models, not because it solved evaluation, but because it compressed a large and messy capability surface into a single memorable score. MMLU General Capability Claims (arXiv)
These benchmarks were not mistakes. They solved real coordination problems. Without GLUE, SuperGLUE, or MMLU, progress claims would have remained anecdotal, incomparable, and much easier to spin. But they also encouraged a habit of mind: treating a public score as a stable stand-in for a capability distribution. The later criticism of benchmarking is best read not as a rejection of benchmarking, but as a rejection of that stronger inference. Benchmark Lottery Construct Validity (arXiv)
The repair cycle: adversarial, dynamic, and live benchmarks
The first explicit repair was adversarial and dynamic data collection. ANLI used a human-and-model-in-the-loop process to create difficult natural-language inference examples that contemporary systems failed on, rather than assuming that a fixed test set would stay challenging. Dynabench generalized that idea into a platform: benchmark creation should itself be dynamic, with humans probing models and refreshing the distribution in response to model progress. ANLI Dynabench Dynamic Benchmarks (arXiv)
A second repair was theory-driven benchmarking. François Chollet’s ARC proposal argued that benchmark success can be purchased with data and priors and therefore is not equivalent to intelligence; the right target is something closer to skill-acquisition efficiency under constrained prior knowledge and experience. ARC-AGI is therefore not just “another hard benchmark.” It is an explicit objection to equating benchmark skill with general intelligence, and it remains important because it makes its philosophical assumptions unusually visible. ARC-AGI Skill Acquisition Efficiency Intelligence versus Skill (arXiv)
A third repair was expert-curated frontier difficulty. GPQA was built from graduate-level “Google-proof” questions written and validated by domain experts in biology, physics, and chemistry. The design goal was not dynamism but resistance: if retrieval and generic exam heuristics can cheaply solve the task, the benchmark is not buying much discriminative power at the frontier. GPQA Expert-Curated Benchmarks (arXiv)
A fourth repair was execution-based realism. SWE-bench moved from short-form answers to repository-grounded software tasks, asking systems to resolve real GitHub issues against real codebases. That was a conceptual shift as much as a benchmark shift: the model was no longer being asked whether it “knows” software engineering facts, but whether it can act in a constrained environment and produce a patch that passes tests. Its follow-on variants, including SWE-bench Verified and SWE-bench-Live, reflect how quickly agentic coding evaluation inherits the same maintenance cycle as earlier benchmarks. SWE-bench Agentic Coding Execution-Based Evals (arXiv)
A fifth repair was freshness itself. LiveCodeBench continuously collects new coding problems from programming contests to reduce contamination and broaden the task surface beyond one-shot code generation. LiveBench generalizes the same instinct beyond coding: frequently updated questions, objective grading, and ongoing refresh are treated as first-class design goals rather than afterthoughts. LiveCodeBench LiveBench Benchmark Freshness (arXiv)
The through-line is that each generation of benchmarks is usually a reaction to a specific way the previous generation failed: too static, too easy to contaminate, too coarse, too short-horizon, too detached from real tooling, or too quickly saturated. The field does not move from benchmarking to “no benchmarks.” It moves from one repair attempt to another. Benchmark Churn Benchmark Maintenance (arXiv)
| Phase | Representative benchmarks | Design move | What it bought | What it still did not solve |
|---|---|---|---|---|
| Static multitask benchmarking | GLUE, SuperGLUE, MMLU | Fixed public datasets with leaderboards | Shared comparison, field coordination, legible progress | Saturation, contamination, and weak connection from score to deployment behavior |
| Adversarial/dynamic benchmarking | ANLI, Dynabench | Humans adapt data collection to model failures | Harder distributions and longer discriminative life | Eventually still partial, gameable, and in need of refresh |
| Theory-driven benchmarking | ARC-AGI | Makes priors and generalization assumptions explicit | Better resistance to pure memorization and “skill-only” readings | Ongoing debate about what broader claims it licenses |
| Expert-curated frontier benchmarking | GPQA | Domain-expert, web-resistant questions | Stronger discrimination at the frontier | Still mostly short-horizon and exam-like |
| Execution-based agent benchmarking | SWE-bench, Verified | Real tasks in real repos with executable outcomes | Higher realism and better connection to workflows | Harness, infra, and grader confounds become central |
| Live/freshness-oriented benchmarking | LiveCodeBench, LiveBench, SWE-bench-Live | Continuous or frequent refresh | Lower contamination and renewed headroom | Continuous maintenance burden and new leakage channels |
The periodization, “what it bought,” and “what it still did not solve” columns are synthesis from the benchmark papers and validity literature rather than a taxonomy taken from a single source. (arXiv)
Contamination, saturation, and validity
Contamination
Contamination Benchmark Contamination is the simplest failure mode to explain and the easiest to underestimate. Open benchmarks are discoverable; modern training corpora are vast; benchmark items or close paraphrases can leak into pretraining, instruction tuning, tool traces, synthetic data, or prompt libraries. The result is not merely unfairness. It is a change in what the score means: part of the measured quantity becomes exposure. (arXiv)
This is why refresh papers around MMLU matter. MMLU-Pro was motivated by plateauing performance and prompt sensitivity on the original MMLU, and it makes the task more reasoning-heavy and less brittle to formatting. MMLU-CF goes further by treating contamination as structural: open-source benchmarks combined with broad training data make leakage increasingly inevitable, so the design response is a public validation set plus a closed test set. MMLU-Pro MMLU-CF (arXiv)
But contamination is not solved by secrecy alone. The recent saturation literature argues that private or held-out test data do not systematically prevent loss of discriminative power. That is an important corrective because it separates two problems that are often conflated: leakage is one way to destroy benchmark meaning, but even uncontaminated tests can stop differentiating if the distribution is too narrow or too thoroughly optimized against. Benchmark Saturation Private Test Sets (arXiv)
Even “live” evaluation has contamination channels. Anthropic reported contamination in BrowseComp because answers to some benchmark questions leaked onto the public web after benchmark creation, contaminating nine examples out of 1,266. Live or web-grounded evaluation reduces some forms of leakage and creates others. BrowseComp Live Benchmark Leakage (Anthropic)
Saturation
The most useful current definition of saturation is loss of discriminative power among frontier systems. That framing matters because a benchmark can be saturated without every model scoring near 100%, and a benchmark can be highly solved yet still remain scientifically useful if what it measures is still the intended construct. The problem is not “the score got high”; the problem is “the score stopped telling you enough.” Benchmark Saturation Discriminative Power (arXiv)
The new saturation study is important precisely because it turns a familiar complaint into something measurable. Its headline result is severe: many widely used benchmarks already show high or very high saturation, with older benchmarks especially vulnerable. Just as important, the paper reports that expert-curated and dynamic or adversarial benchmarks appear more resistant than generic public test sets. That does not make them permanent solutions, but it explains why the field keeps rebuilding in those directions. (arXiv)
The same pattern now appears inside safety evaluation. Anthropic’s transparency materials note that baseline safety evals can become near-perfect, limiting their usefulness for detecting improvement or degradation, which is why the lab introduced higher-difficulty benign-request and edge-case evaluations. OpenAI’s more recent deployment safety reporting makes a parallel move: standard jailbreak and other older safety sets became relatively saturated, so the company added harder production-style benchmarks with more challenging prompts. Safety Evaluation Benchmark Refresh (Anthropic)
There is no single paper proving that benchmarks globally lose discriminative power “faster than” new ones are created. That stronger claim is an inference from repeated benchmark churn, not an established statistic. But the pattern itself is unmistakable: GLUE gave way to SuperGLUE; MMLU generated MMLU-Pro and MMLU-CF; SWE-bench generated Verified and Live variants; baseline safety sets spawned harder or more production-like successors. Benchmark replacement is now a recurring maintenance burden rather than an occasional reset. Benchmark Churn (arXiv)
Construct validity
The hardest problem is construct validity Construct Validity: whether the benchmark score actually measures the thing people say it measures. The Benchmark Lottery paper argued that apparent model superiority can depend heavily on benchmark choice and benchmark-specific affordances. “Winning the benchmark” is therefore not equivalent to “being the better system” in any global sense. (arXiv)
That point has only strengthened. AI and the Everything Benchmark argues that broad, influential AI benchmarks are often used as stand-ins for “general” progress without sufficient construct validity. A recent systematic review of 445 LLM benchmarks identifies recurring design and reporting patterns that undermine strong validity claims, and an interdisciplinary review of benchmark trustworthiness catalogs problems ranging from inadequate documentation to bias, contamination, noisy measurement, and the mismatch between one-time testing and interactive human-AI systems. External Validity Measurement Validity (arXiv)
The right conclusion is not that benchmarks are useless. It is that benchmark scores are claims about a measurement instrument, not direct observations of intelligence, reliability, or safety. That is the measurement problem in one sentence. AI Measurement Problem (arXiv)
Goodhart’s Law applied to AI benchmarks
Goodhart’s Law Goodhart’s Law for AI is the cleanest conceptual language for what goes wrong when benchmarks become central. In AI evaluation, it means that once a score becomes a target for research prestige, product iteration, or release gating, the score often becomes easier to improve than the underlying capability it was meant to proxy. That is not a metaphorical risk; the critical literature now contains direct examples of metrics being inflated without corresponding gains in the intended construct. (arXiv)
A useful way to think about benchmark Goodharting is to split it into four layers. This taxonomy is a synthesis from the cited literature rather than a standard named framework. First is data Goodhart: contamination and distribution learning raise the score because the test has become familiar. Second is metric Goodhart: models optimize the grader or metric, sometimes by exploiting surface regularities. Third is harness Goodhart: agent benchmarks improve because the scaffold, budget, retry policy, or evaluation runtime improves rather than because the underlying model became proportionally more capable. Fourth is deployment Goodhart: teams optimize an internal eval bank so aggressively that it drifts away from actual user value or real-world failure modes. Benchmark Contamination LLM-as-Judge Agent Harness Deployment Drift (arXiv)
The harness layer is especially important for current agent systems. Anthropic’s infrastructure-noise analysis shows that configuration choices can swing coding benchmark scores by several percentage points, sometimes more than the headline gaps between competing systems, and that a nontrivial fraction of tasks can fail for reasons unrelated to model ability. When that is true, two systems with different runtime budgets or infra assumptions are not really taking the same test. Agent Harness Infrastructure Noise (Anthropic)
Goodhart pressure does not imply that optimization is illegitimate. The entire point of engineering is to optimize. The lesson is narrower: any sufficiently important benchmark has to be treated as a living proxy that needs refresh, adversarial maintenance, and contact with production reality. A benchmark that cannot survive optimization pressure cannot remain a stable reference object. Benchmark Maintenance Adversarial Evaluation (Dynabench)
Capability evaluation versus safety evaluation
Capability evals ask whether the model or system can do a target task under specified conditions. Safety evals ask whether the same system creates unacceptable risk under relevant threat models or deployment conditions. The two overlap, but they are not reducible to each other. A model can become more capable while also becoming more dangerous, more over-eager, more vulnerable to prompt injection, or harder to assess. Capability Evaluation Safety Evaluation Prompt Injection Reward Hacking (OpenAI)
OpenAI’s Preparedness Framework makes this distinction explicit by defining tracked categories only when the lab believes evals can closely track severe-harm potential. Its tracked categories include biological and chemical risk, cybersecurity, and AI self-improvement, while research categories include longer-horizon and more speculative concerns such as long-range autonomy, autonomous replication and adaptation, and sandbagging. That is not “just benchmarking.” It is risk measurement under governance constraints. Preparedness Framework Long-Range Autonomy Sandbagging (OpenAI)
Anthropic’s current safety reporting reflects a similar split. The Transparency Hub and system-card materials summarize safety evaluations around prompt injection, ambiguous context, over-eagerness or reward-hacking-like behavior, and agentic misuse scenarios with tool access. The documents are explicit that alignment assessment is preliminary and evolving, and they also note a subtle measurement problem: if models behave differently when they suspect they are being tested, then the eval itself may stop measuring ordinary deployed behavior. Anthropic Transparency Evaluation Awareness (Anthropic)
That last point is easy to miss. Safety evaluation is not just capability benchmarking with a moral vocabulary. It has its own construct-validity problems, and in some cases those problems are worse, because the system may be strategically responsive to the evaluation context. A near-perfect refusal score on a baseline set does not prove robust safety; it may only prove competence on a now-familiar surface distribution. Safety Benchmark Saturation (Anthropic)
| Dimension | Capability eval | Safety eval | Production/custom eval |
|---|---|---|---|
| Core question | Can the system do the task? | Does the system create unacceptable risk or fail under threat models? | Does this exact application behave acceptably for our users and workflow? |
| Typical data | Public benchmark tasks, domain tasks, executable tasks | Adversarial prompts, harmful or ambiguous scenarios, preparedness tasks, prompt-injection or over-eagerness probes | Production traces, historical failures, edge cases, expert-written cases, domain-specific samples |
| Common grading target | Correctness, pass rate, task completion | Harmful uplift, refusal or over-refusal, robustness, safeguard adherence | Outcome success, trace quality, cost, latency, tool use, user-aligned rubrics |
| Typical examples | MMLU, GPQA, SWE-bench | Preparedness categories, prompt injection, ambiguous context, agentic misuse evals | OpenAI app-specific tests, Anthropic task-specific success criteria, private regression suites |
| Main failure mode | Saturation and contamination | Distribution mismatch and saturation on standard safety sets | Grader drift, proxy misalignment, local over-optimization |
This table is a synthesis of the cited benchmark papers, OpenAI’s eval documentation, the Preparedness Framework, and Anthropic’s current safety and evaluation guidance. (arXiv)
From leaderboards to custom eval suites in production
The strongest evidence for the center-of-gravity shift is not a paper; it is the vendor documentation itself. OpenAI’s evaluation best-practices page explicitly distinguishes public benchmarks from application-specific tests and frames evals as the way to test AI systems in production-like conditions, precisely because generative outputs are variable and traditional software testing is not enough. In other words, the public benchmark is still useful, but the real operational object is the custom suite. Production LLM Evals Custom Evals (OpenAI Developers)
Anthropic’s guidance makes the same move from a different angle. It tells teams to define success criteria that are specific and measurable, notes that even ethics and safety dimensions can be operationalized, and warns that most real use cases need multidimensional evaluation rather than a single scalar score. That is a direct break from benchmark monoculture. Multidimensional Evaluation Success Criteria (Claude API Docs)
This is also where the semantics of “eval” changed. In public discourse, an eval still often means “a benchmark score.” In engineering practice, it increasingly means “a bank of cases with grading logic.” OpenAI’s openai/evals framework is explicit about this: it is a framework for evaluating LLMs or LLM-based systems, with support for existing registries, custom evals, and private evals over the user’s own data. The emphasis is not on a single universal leaderboard but on instrumenting the thing you are actually shipping. openai/evals Private Evals (GitHub)
Even the surrounding tooling points in the same direction. OpenAI’s Prompt Optimizer uses small labeled datasets whose rows can include grader results, human annotations, or critiques, and it treats optimization as an eval-mediated loop rather than a prompt-writing art form. Anthropic’s Evaluation Tool lives inside the Claude Console’s evaluation workflow and includes built-in prompt-generation support, again signaling that prompt iteration is now expected to happen against explicit tests rather than by intuition alone. Prompt Optimization Prompt Iteration (OpenAI Developers)
There is a methodological lesson here. Public benchmarks are best interpreted as external reference instruments: good for rough capability mapping, public comparison, and falsifying obviously weak systems. Custom eval suites are decision instruments: they are what teams use to approve a release, catch regressions, compare prompting or model choices, and monitor whether the product is drifting away from intended behavior. Treating the first as a substitute for the second is exactly the mistake current practice is trying to correct. Reference Instrument Decision Instrument (OpenAI Developers)
Evals as tests: the eval-driven development pattern
“Eval-driven development” Eval-Driven Development is not yet a settled academic term, but it is already a recognizable engineering pattern. OpenAI’s “Working with Evals” guide explicitly likens the process to behavior-driven development: describe the task as an eval, run it on test inputs, inspect the results, and iterate. The cookbook on eval-driven system design pushes the same idea further, describing evals as the core process for building a production-grade autonomous system. (OpenAI Developers)
That framing is more important than it may first appear. In classic software engineering, a test specifies acceptable behavior and protects it against regression. In AI engineering, a custom eval increasingly plays the same role, except that the system under test is stochastic and the acceptance criterion may itself involve model-based grading, human review, or environment-level success conditions. OpenAI’s agent-evals material therefore emphasizes reproducible agent evaluation and workflow-level or trace grading, not just final-answer comparison. Agent Evals Trace Grading (OpenAI Developers)
Anthropic’s agent-evals guidance makes the same point in plainer language: good evals let teams ship with confidence; bad or absent evals trap them in reactive loops. It also offers a practical design principle that many benchmark papers implicitly ignore: grading the exact tool path is often too rigid, so evaluators should frequently score the produced outcome rather than a single canonical sequence of intermediate actions. That is a deeply software-testing way to think about agents. Outcome-Based Grading Reactive Loop (Anthropic)
This pattern also changes what counts as a “good” eval. OpenAI and Anthropic both push teams toward explicit success criteria, production or historical data, synthetic augmentation where needed, automated grading where possible, and continuous reruns over time. Anthropic adds an operationally important warning: the full picture includes production monitoring, user feedback, A/B tests, transcript review, and systematic human evaluation. Offline synthetic evals are necessary; they are not enough. Continuous Evaluation Transcript Review A/B Testing (OpenAI Developers)
The statistical side matters too. Anthropic’s statistical guidance reminds practitioners that the observed score is not the object of interest; the object is the expected score over the broader task distribution. It also documents how naive uncertainty estimates can badly mislead—for example, clustered standard errors can be several times larger than naive ones when an eval has internal structure. That is a quiet but important point: many leaderboard deltas and agent-eval improvements are smaller than the uncertainty introduced by the benchmark or harness design itself. Statistical Power Confidence Intervals Clustered Errors (Anthropic)
The right mental model is therefore not “evals replace judgment.” It is “evals structure judgment.” They create a reproducible conversation between the system, the grader, and the engineers. The strongest teams appear to be converging on that model: evals as tests, not as oracles. LLM-as-Judge Human-in-the-Loop Evaluation (OpenAI Developers)
The open problem: long-horizon agent behavior
Long-horizon agent evaluation is where the older benchmark paradigm breaks most visibly. Benchmarks like WebArena, OSWorld, and GAIA try to measure something closer to what people now actually want from agents: sustained interaction with tools and environments over many steps. But as soon as you do that, the measured object stops looking like a pure model and starts looking like a coupled system. Long-Horizon Agents Time-Horizon Benchmarks (arXiv)
The benchmark results themselves show why this frontier matters. WebArena was created as a realistic, reproducible web environment because many prior web-agent evaluations relied on toy or synthetic settings. OSWorld includes hundreds of real computer-use tasks across applications and operating systems and reports a very large human-model gap. GAIA similarly shows that tasks designed to be conceptually easy for humans but operationally demanding for models remain difficult even for strong assistants with tool access. WebArena OSWorld GAIA (arXiv)
But the measurement difficulty rises just as fast as the realism. WebArena Verified showed that the original benchmark substantially underestimated agent capability because of task ambiguity and evaluator fragility; its re-evaluation reduced false negatives by 11.3 percentage points through task auditing and stronger verification procedures. That is a direct warning against taking long-horizon benchmark scores at face value without first interrogating the grader. WebArena Verified Evaluator Fragility (OpenReview)
The infrastructure itself also becomes part of the test. Anthropic’s infrastructure-noise analysis shows that resource configuration, runtime setup, and platform failures can materially swing benchmark results in agentic coding. Their harness-design work adds a complementary point: frontier agent performance depends on how generation, evaluation, context resets, and feedback loops are structured. At that point, saying “the model scored X” can hide the real causal story. Agent Harness Infrastructure Noise Evaluation Runtime (Anthropic)
Contamination and benchmarking pressure do not disappear here; they mutate. The BrowseComp contamination report shows that even web-grounded benchmarks can leak because answers propagate into the public internet. That matters especially for long-horizon agent work, because systems increasingly retrieve from the very environments that benchmark designers assume will remain fresh. BrowseComp Retrieval Contamination (Anthropic)
There is also a deeper conceptual problem. The METR line of work argues that one promising way to summarize frontier capability is by task-completion time horizon: roughly, how long a human task can the AI reliably complete. That is an appealing move because it converts many heterogeneous tasks into a continuous measure. But it is not yet a solved answer to long-horizon evaluation. Time horizon depends on the environment, the scaffold, the human baseline used for calibration, the decomposition strategy, and the success criterion. It is a promising research program, not a finished measurement standard. Time Horizon Continuous Capability Measures (METR)
DeepPlanning sharpens a related critique: many “long-horizon” benchmarks still reward local stepwise competence more than genuine global planning under constraints. That is exactly the kind of construct-validity failure that AI benchmarking has already seen in other domains. A benchmark can look like it measures long-range agency while mostly measuring local reasoning plus a forgiving scaffold. DeepPlanning Planning Benchmarks (arXiv)
The honest conclusion is stark. There is still no MMLU-like consensus instrument for long-horizon agents, and perhaps there should not be. The object of evaluation is too entangled: model, tool interface, retry policy, context-management scheme, evaluator, compute budget, and environment stability all materially affect the observed score. Long-horizon evaluation is therefore not just a harder benchmark problem. It is a full systems-measurement problem. AI Measurement Problem Agent Systems (arXiv)
What seems true now
A few conclusions look relatively robust.
First, public benchmarks still matter, but mainly as shared external reference points. They are valuable for rough capability mapping, public comparability, and detecting obviously weak or obviously strong systems. They are much less reliable as acceptance tests for production behavior. AI Benchmarks Production LLM Evals (arXiv)
Second, the field is on a treadmill of benchmark renewal. Dynamic, expert-curated, theory-driven, and live benchmarks all buy time against saturation and contamination, but none of them escape the basic logic that important scores will be optimized against. Benchmark design is now a continuing adversarial process, not a one-time publication event. Adversarial Evaluation Benchmark Maintenance (Dynabench)
Third, production practice has already moved beyond leaderboard thinking. The most useful current evidence is not rhetorical but operational: OpenAI and Anthropic both tell practitioners to define explicit success criteria, build custom suites, grade outcomes and traces, inspect failures, and rerun evals continuously. That is software-testing language, because in production that is effectively what evals have become. Eval-Driven Development Regression Testing for AI Systems (OpenAI Developers)
Fourth, safety evaluation is not a subset of capability evaluation. It has its own benchmarks, failure modes, adversaries, and construct-validity problems. Near-perfect scores on baseline refusals or standard jailbreak sets should now be read with suspicion unless accompanied by fresher and more deployment-relevant evidence. Safety Evaluation Prompt Injection Ambiguous Context (Anthropic)
Fifth, long-horizon agent evaluation remains genuinely open. The frontier difficulty is no longer just creating hard questions; it is defining what the system under test actually is and preserving measurement meaning when the harness is part of the capability. If the 2018–2023 era was about benchmarking models, the 2024–2026 era increasingly looks like benchmarking systems. Long-Horizon Agents Agent Harness System Evaluation (arXiv)
References and source guide
1 GLUE paper — Wang et al., “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.” (arXiv)
2 SuperGLUE paper — Wang et al., “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems.” (arXiv)
3 MMLU paper — Hendrycks et al., “Measuring Massive Multitask Language Understanding.” (arXiv)
4 ANLI paper — Nie et al., “Adversarial NLI: A New Benchmark for Natural Language Understanding.” (arXiv)
5 Dynabench paper/platform — Kiela et al., “Dynabench: Rethinking Benchmarking in NLP”; platform materials. (Dynabench)
6 ARC / Chollet — Chollet, “On the Measure of Intelligence”; ARC prize materials. (arXiv)
7 GPQA paper — Rein et al., “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.” (arXiv)
8 SWE-bench paper — Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (arXiv)
9 SWE-bench refreshes — SWE-bench Verified and SWE-bench-Live materials. (arXiv)
10 Live coding benchmarks — LiveCodeBench and LiveBench. (arXiv)
11 OpenAI evaluation best practices — OpenAI developer docs on eval categories, datasets, and production testing. (OpenAI Developers)
12 OpenAI working with evals — behavior-driven-development framing for eval iteration. (OpenAI Developers)
13 OpenAI prompt optimizer — eval-mediated prompt optimization using labeled rows, grader results, and annotations. (OpenAI Developers)
14 OpenAI eval-driven system design — cookbook on evals as the core process for production-grade autonomous systems. (OpenAI Developers)
15 OpenAI agent evals guidance — reproducible evaluations and trace grading for agents. (OpenAI Developers)
16 openai/evals framework — framework and registry for custom and private evals. (GitHub)
17 Anthropic success-criteria guidance — defining measurable, multidimensional evaluation criteria. (Claude API Docs)
18 Anthropic Evaluation Tool — evaluation workflow and built-in prompt-generation support in Claude Console. (Claude API Docs)
19 Anthropic agent-evals guidance — “Demystifying evals for AI agents.” (Anthropic)
20 Anthropic statistical guidance — statistical approach to model evaluations, expected score versus observed score, structured uncertainty. (Anthropic)
21 Anthropic harness and infrastructure guidance — infrastructure noise and harness design for long-running applications. (Anthropic)
22 OpenAI safety/governance docs — Preparedness Framework and recent system-card materials. (OpenAI)
23 Anthropic transparency and safety reporting — Transparency Hub and system-card style reporting on safety evaluations. (Anthropic)
24 Contamination literature — “Investigating Data Contamination in Modern Benchmarks for Large Language Models.” (arXiv)
25 MMLU refreshes — MMLU-Pro and MMLU-CF. (arXiv)
26 Benchmark saturation study — recent formalization of benchmark saturation and discriminative-power loss. (arXiv)
27 Benchmark validity literature — Benchmark Lottery; AI and the Everything Benchmark; systematic and interdisciplinary benchmark-validity reviews. (arXiv)
28 Goodhart-style critique — benchmark and metric gaming in AI evaluation. (arXiv)
29 Long-horizon web and computer-use benchmarks — WebArena, WebArena Verified, OSWorld, GAIA. (arXiv)
30 Long-horizon measurement programs — BrowseComp contamination note, METR time horizon work, DeepPlanning. (Anthropic)
Companion entries
Core theory: AI Measurement Problem, Construct Validity, Benchmark Saturation, Benchmark Contamination, Benchmark Lottery, Goodhart’s Law for AI, External Validity, Measurement Validity
Benchmark families: GLUE, SuperGLUE, MMLU, MMLU-Pro, MMLU-CF, ANLI, Dynabench, ARC-AGI, GPQA, SWE-bench, LiveCodeBench, LiveBench
Practice: Production LLM Evals, Custom Evals, Eval-Driven Development, Regression Testing for AI Systems, Prompt Optimization, Agent Evals, Trace Grading, LLM-as-Judge, Human-in-the-Loop Evaluation
Safety and governance: Capability Evaluation, Safety Evaluation, Preparedness Framework, Prompt Injection, Ambiguous Context, Reward Hacking, Sandbagging, Long-Range Autonomy
Agents and long-horizon evaluation: Long-Horizon Agents, Agent Harness, Infrastructure Noise, WebArena, OSWorld, GAIA, Time-Horizon Benchmarks, Planning Benchmarks