Abstract. LiveBench is best understood as a benchmark-lifecycle proposal, not merely another leaderboard: it combines Time-Gated Evaluation, broad task coverage, and objective/verifiable scoring to slow the decay that afflicts static LLM benchmarks. Its central claim is that evaluation sets for frontier models must be refreshed, versioned, and auditable, while still avoiding the fragility of LLM-as-judge scoring where possible. This article covers the LiveBench design, empirical results, comparisons to LiveCodeBench, MMLU, and MMLU-Pro, and the open question of whether refreshed multi-domain benchmarks become the default pattern for AI evaluation.
Coverage note: verified through May 19, 2026.
LiveBench: Time-Gated Multi-Domain Evaluation
1. Why LiveBench exists
LiveBench responds to a concrete failure mode in LLM Evaluation: once a benchmark’s questions are public, they can appear in pretraining corpora, fine-tuning sets, evaluation harnesses, blog posts, model cards, and synthetic distillation data. White et al. frame this as test-set contamination, arguing that static internet-published benchmarks can become unreliable as newer models are trained on ever-larger web and code corpora. Their proposed countermeasure is not secrecy alone, but a rolling benchmark whose questions are drawn from recent sources or generated procedurally, scored by objective ground truth, and spread across multiple capability domains. arXiv
The subtle but important terminology shift is from “contamination-free” to “contamination-limited.” Early LiveBench artifacts used the stronger label, but the ICLR/OpenReview version titles the work “Contamination-Limited,” which is more defensible: any public benchmark can eventually be absorbed into future training data, so the operational defense is versioning, recency, delayed release, and continuing replacement rather than permanent immunity. LiveBench
LiveBench’s practical thesis is that Benchmark Contamination is a lifecycle problem. A benchmark should not be treated as a one-time dataset release; it should be maintained like production infrastructure, with release dates, task retirement, scoring code, model-answer artifacts, and explicit provenance.
2. Core design
LiveBench has three design commitments: refreshed questions, verifiable scoring, and multi-domain coverage. The paper describes tasks based on recent math competitions, arXiv papers, news articles, datasets, and more difficult variants of earlier benchmark tasks such as Big-Bench Hard, AMPS, and IFEval. The benchmark originally covered math, coding, reasoning, language comprehension, instruction following, and data analysis, and the public repository summarizes the same principles: recent or frequently refreshed sources, objective ground-truth answers, and diverse tasks across six categories. arXiv
| Design axis | LiveBench choice | Why it matters | Tradeoff |
|---|---|---|---|
| Contamination defense | Rolling releases from recent sources and procedural/task variants | Makes it harder for a model to win by memorizing old public test items | Requires continuous maintenance and version-aware comparisons |
| Scoring | Verifiable answers, execution tests, parsers, or rule checks rather than broad LLM judging | Reduces judge bias and improves reproducibility | Excludes many open-ended tasks that matter in production |
| Coverage | Math, coding, reasoning, language, instruction following, data analysis | Prevents single-skill overfitting and exposes uneven model profiles | Each domain is necessarily thinner than a domain-specialized benchmark |
| Difficulty management | Newer and harder tasks over time; ambiguous/saturated tasks can be removed | Preserves signal as frontier models improve | Scores across releases are not always directly comparable |
| Transparency | Public code, questions, and model answers, with release tracking | Enables replication and error analysis | Public release creates future contamination pressure |
The original paper announced monthly additions and updates. The current public site is indexed as saying that questions are updated regularly so the benchmark completely refreshes every six months; Inspect Evals’ LiveBench implementation similarly notes a six-month full refresh and delayed release of the most recent questions. This matters because the benchmark’s anti-contamination story depends less on the word “monthly” than on sustained, versioned replacement. arXiv+2LiveBench+2
3. Task taxonomy
LiveBench is multi-domain, but it is not a “general intelligence” benchmark in the loose leaderboard sense. It is a collection of verifiable task families chosen to stress reasoning, procedural robustness, and resistance to stale training data.
| Category | Original task sources / examples | What it probes | Scoring style |
|---|---|---|---|
| Math | Recent AMC/AIME/USAMO/IMO/SMC-style problems and harder AMPS-style questions | Symbolic reasoning, algebraic manipulation, competition math | Exact or normalized answer matching |
| Coding | LiveCodeBench-derived generation tasks and code-completion tasks; later refreshed toward real-world library use and agentic coding | Algorithmic problem solving, code synthesis, completion, and later issue-resolution workflows | Execution tests / harnessed evaluation |
| Reasoning | Harder Web-of-Lies variants, Zebra puzzles, spatial reasoning, later theory-of-mind and navigation tasks | Multi-step logical consistency and constraint tracking | Parsed final answers |
| Language comprehension | Connections-style word puzzles, typo fixing in recent arXiv abstracts, movie-synopsis unscrambling | Linguistic reasoning beyond factual recall | String, fuzzy, or ordering-based metrics |
| Instruction following | Paraphrase, simplification, summarization, and story generation over recent news articles under checkable constraints | Whether a model follows exact output constraints while preserving task content | Rule-level instruction adherence checks |
| Data analysis | Table reformatting, table-join prediction, column type annotation; later refreshed and extended | Practical data wrangling and schema reasoning | Structured-output comparison |
The paper states that the initial benchmark contained 18 tasks across six categories; its task examples include recent math competitions, LiveCodeBench-derived coding tasks, Big-Bench-Hard-style reasoning variants, recent movie and arXiv language tasks, Guardian-news instruction-following tasks, and Kaggle/Socrata-derived data-analysis tasks. LiveBench
The task set is explicitly mutable. The changelog records removals, replacements, and additions: a house-traversal task was removed after ambiguity and saturation issues; spatial reasoning was added; math sources were refreshed; coding questions were updated; agentic coding was introduced in 2025; the agentic coding harness later switched from SWE-Agent to Mini-SWE-Agent; and new reasoning/math/data-analysis tasks were added in late 2025 and early 2026. GitHub
That mutability is not incidental. It is LiveBench’s methodological contribution. Static benchmarks ask, “How did models score on this fixed test?” LiveBench asks, “How should an evaluation keep producing signal after models, training corpora, and benchmark-aware optimization have moved on?”
4. Methodology contribution
4.1 Time-gating as an evaluation primitive
Time-Gated Evaluation means that benchmark items are anchored to release dates or fresh information sources, so evaluators can ask whether a model plausibly saw the item before training. LiveBench does this across domains rather than only in programming contests or current-events tasks. The paper’s contamination defense combines recent sources, procedural generation, and continuing replacement; the current ecosystem adds release-date parameters and delayed public release in downstream implementations. arXiv
This is weaker than a private benchmark but stronger than a static public one. A private benchmark can be harder to contaminate but less reproducible. A public rolling benchmark can be reproduced, audited, and improved, but must keep moving because yesterday’s fresh questions become tomorrow’s training data.
4.2 Objective scoring over LLM judging
LiveBench deliberately avoids general LLM-as-judge scoring. White et al. argue that LLM judges can be biased and unreliable on hard reasoning and math tasks; the paper reports that GPT-4-Turbo pass/fail judgments can reach high error rates on challenging problems. The design response is to include only tasks with verifiable answers or checkable constraints, even when the task surface looks open-ended. LiveBench
This has a philosophical cost. Many important production tasks—email drafting, strategic synthesis, negotiation, taste-sensitive writing, design review—do not have a single objective answer. LiveBench therefore occupies a precise niche: it is a benchmark for capability slices that can be made hard, recent, and objectively checkable.
4.3 Multi-domain discrimination
The six-domain structure is central. A code-only benchmark can be excellent for programming, and an exam benchmark can be excellent for academic knowledge, but broad model selection often fails when one scalar hides uneven profiles. LiveBench’s original Figure 1 and result tables show that category-level rankings differ: a model can be strong in instruction following but weaker in coding, or strong in coding but less balanced across language and reasoning. LiveBench
This matters for Model Selection. If a team is choosing a general assistant model, a single aggregate score is not enough. LiveBench’s design encourages reading the per-category breakdown before the overall score.
5. Empirical results
5.1 Original 2024 frontier picture
In the original LiveBench paper’s 49-model table, Claude 3.5 Sonnet 20240620 led with an overall LiveBench score of 61.2, followed by GPT-4o 2024-05-13 at 55.0 and GPT-4-Turbo 2024-04-09 at 53.0. The same table shows strong category variation: Claude 3.5 Sonnet’s score was especially high in instruction following and reasoning, while GPT-4o was close on instruction following but behind on coding and reasoning. LiveBench
The paper’s broad empirical claim was that LiveBench was hard enough to avoid immediate saturation: top models were below roughly 65% in the original version, while the later OpenReview/arXiv abstract describes top models below 70% after expanded evaluations. That shift is not a contradiction so much as evidence of the benchmark’s moving context: the model pool, question releases, and benchmark title evolved between the first arXiv artifact and the ICLR version. LiveBench+2arXiv+2
5.2 Recent leaderboard picture
The recent official LiveBench leaderboard snippets show frontier scores clustered in the mid-to-high 70s rather than near-perfect saturation; one indexed snapshot lists Claude 4.7 Opus Thinking xHigh Effort at 76.91 and Claude 4.6 Opus Thinking High Effort at 76.33, while another official snippet includes GPT-5.1 High at 72.04, Qwen 3.6 Plus at 70.85, GPT-5 Pro at 70.48, and GLM 5.1 at 70.18. The exact ordering is a moving target, but the important signal is that LiveBench remains materially below a 90–100% ceiling in official-looking current snapshots. LiveBench
Third-party leaderboard views require caution. LLM Stats’ LiveBench page, last updated May 19, 2026, reports a different small set of 13 models with o3-mini at 84.6%, Qwen3 235B A22B at 77.1%, and Kimi K2-Instruct-0905 at 76.4%, while also marking the listed results as self-reported and unverified. This illustrates a governance issue for rolling benchmarks: once external aggregators copy, filter, or reinterpret benchmark results, “the LiveBench score” can fragment unless release dates, task versions, scoring code, and verification status are preserved. LLM Stats
The right empirical reading is therefore not “model X is universally better than model Y because of one LiveBench score.” It is: LiveBench retains enough headroom and breadth to reveal nontrivial differences among frontier models, but its value depends on reading the version, category breakdown, release date, and evaluation provenance.
6. Comparison to LiveCodeBench
LiveCodeBench is the nearest methodological cousin. It uses the same time-gated philosophy, but it is code-focused: it continuously collects new programming problems from LeetCode, AtCoder, and Codeforces and evaluates code-related capabilities such as generation, self-repair, code execution, and test-output prediction. The LiveCodeBench paper reports hundreds of problems released over time and explicitly discusses contamination, overfitting, and post-cutoff evaluation. arXiv
| Dimension | LiveBench | LiveCodeBench |
|---|---|---|
| Scope | Multi-domain: math, coding, reasoning, language, instruction following, data analysis | Coding-only |
| Time-gating mechanism | Recent sources, procedural/task refreshes, release dates, delayed latest release | Contest problems collected over time, with problem release dates |
| Scoring | Verifiable answers, rule checks, execution tests, structured comparisons | Program execution and task-specific code metrics |
| Best use | Broad model discrimination under contamination pressure | Code-model and programming-agent evaluation |
| Weakness | Harder to maintain consistently across heterogeneous tasks | Narrower capability coverage |
| Relationship | Includes coding but is not primarily a coding benchmark | Supplies a model for how time-gating can work well in one domain |
Recent LiveCodeBench leaderboard views show how a domain-specific time-gated benchmark can scale as a public signal. A May 19, 2026 Price Per Token page sourcing Artificial Analysis data reports 218 evaluated models and a top LiveCodeBench score of 91.7% for Gemini 3 Pro Preview, followed by Gemini 3 Flash Preview at 90.8% and DeepSeek V3.2 Speciale at 89.6%. That high top-end score suggests that even time-gated benchmarks can approach saturation within a narrower domain unless difficulty and time windows keep evolving. Price Per Token
The LiveBench-versus-LiveCodeBench distinction is therefore not “broad is better than narrow.” It is a division of labor. LiveCodeBench gives deeper coding signal; LiveBench gives broader cross-domain signal. A practitioner evaluating developer tools should weight LiveCodeBench or SWE-bench-style tasks heavily; a practitioner evaluating a general reasoning assistant should prefer LiveBench-like multi-domain slices plus task-specific internal evals.
7. Comparison to MMLU and MMLU-Pro
MMLU became the canonical broad academic benchmark because it covered 57 subjects, including elementary math, U.S. history, computer science, law, and other professional or academic areas. Hendrycks et al. designed it to test broad world knowledge and problem solving, and it was extremely useful when frontier models were far from expert-level performance. arXiv
Its weakness is lifecycle, not historical importance. MMLU is static and public. As frontier models improved, MMLU became less discriminative; MMLU-Pro’s authors state that performance on MMLU had begun to plateau and that GPT-4’s March 2023 score left little meaningful room for progress on the original benchmark. arXiv
MMLU-Pro is a stronger static successor. It expands the answer set from four to ten choices, removes trivial or noisy questions, increases reasoning-heavy content, spans 14 domains with over 12,000 questions, and reports a 16–33% accuracy drop compared with MMLU. It also shows lower prompt sensitivity than MMLU and better rewards chain-of-thought-style reasoning. arXiv
| Benchmark | Breadth | Time-gated? | Scoring | Main virtue | Main lifecycle risk |
|---|---|---|---|---|---|
| MMLU | Very broad: 57 subjects | No | Multiple choice | Historical comparability and subject breadth | Saturation and contamination |
| MMLU-Pro | Broad: 14 domains, 12k+ harder questions | No built-in rolling release | Ten-choice multiple choice | Harder and more reasoning-focused than MMLU | Eventually becomes another static public target |
| LiveCodeBench | Narrow but deep coding | Yes | Code execution / task checks | Fresh code signal and cutoff-aware evaluation | Domain specificity and eventual saturation windows |
| LiveBench | Broad multi-domain | Yes / rolling refresh | Verifiable answers and task-specific scoring | Cross-domain signal under contamination pressure | Maintenance burden and version fragmentation |
The marketplace has already adjusted away from pure MMLU reliance. Vellum’s 2026 leaderboard says it features non-saturated benchmarks and excludes outdated benchmarks such as MMLU. Artificial Analysis’ 2026 methodology likewise treats MMLU-Pro as a standalone evaluation rather than part of its current Intelligence Index, and its version history says that MMLU-Pro, LiveCodeBench, and AIME 2025 were removed from the index in v4.0 as the suite shifted toward newer task families. Vellum
This does not make MMLU or MMLU-Pro useless. It means they are now baseline or historical instruments. LiveBench’s contribution is to make “refreshability” a first-class property rather than an afterthought.
8. Practitioner interpretation
For practitioners, LiveBench is most useful when the question is: “Which frontier model is broadly stronger on recent, hard, objectively checkable tasks?” It is less useful when the question is: “Which model will perform best on my company’s workflows, my retrieval corpus, my users, my tool stack, and my error tolerance?”
A practical Evaluation Stack should treat LiveBench as one layer:
| Practitioner need | LiveBench usefulness | Required supplement |
|---|---|---|
| General model comparison | High | Read category breakdowns and version metadata |
| Regression testing after model upgrade | Medium-high | Freeze a release and compare against internal golden tasks |
| Coding assistant selection | Medium | Add LiveCodeBench, SWE-bench, repository-specific tasks |
| Data-analysis assistant selection | Medium | Add internal schemas, messy tables, domain-specific ETL tasks |
| Instruction-following robustness | Medium | Add policy, formatting, and workflow-specific constraints |
| Creative writing, email, strategy, design taste | Low-medium | Human or rubric-based evals; pairwise preference tests |
| Multilingual or multimodal deployment | Limited | Use multilingual/multimodal benchmarks and local user data |
The key anti-pattern is leaderboard monoculture. LiveBench is valuable because it makes contamination harder and preserves some cross-domain headroom, not because it magically measures “the best model.” Its own design choices exclude tasks that cannot be objectively verified. That exclusion is principled, but it is still an exclusion.
9. Will time-gated multi-domain benchmarks become dominant?
The case for dominance is strong. Training corpora increasingly absorb public benchmark artifacts, model providers optimize for visible leaderboards, and users care about current capability rather than 2020-era exam performance. A refreshed benchmark with public scoring code and release dates is structurally better aligned with frontier evaluation than a fixed benchmark that remains unchanged for years.
The case for niche status is also strong. A benchmark like LiveBench is expensive to maintain, and every update can introduce new ambiguity, new parsers, new scoring edge cases, and new comparability problems. Inspect Evals’ documentation explicitly notes that LiveBench updates can break downstream code when new task types require different scoring methods. UK Government BEIS
The most likely outcome is neither total dominance nor irrelevance. Time-gated evaluation will become a dominant pattern for public frontier benchmarks where objective answers are feasible, while LiveBench-style multi-domain leaderboards will remain one component of a larger evaluation portfolio. Domain-specific time-gated benchmarks, private enterprise evals, agentic task suites, human-preference arenas, and safety evaluations will coexist.
In other words: the pattern will dominate; any single benchmark probably will not.
10. Will LiveBench’s refresh cadence continue or fragment?
LiveBench’s visible history supports both optimism and caution. The project did not stop at the initial 2024 release: its changelog shows multiple updates through January 2026, including refreshed math, language, coding, reasoning, instruction-following, data-analysis, and agentic-coding tasks. It also records active task retirement and harness changes, which are signs of a living benchmark rather than a frozen dataset. GitHub
The caution is that “monthly refresh” is operationally demanding. The public messaging has shifted toward full refresh over six months, downstream implementations expose release-date options, and third-party leaderboards may report partial, self-reported, or differently filtered result sets. LiveBench+2UK Government BEIS+2
Fragmentation can occur at several layers:
| Fragmentation layer | Example failure mode | Mitigation |
|---|---|---|
| Dataset version | Model A evaluated on one release, Model B on another | Always publish release date and task manifest |
| Scoring harness | Forks patch parsers or execution environments differently | Pin scorer versions and containers |
| Leaderboard ingestion | Aggregators mix verified and self-reported scores | Mark verification status and source |
| Task evolution | New tasks change the meaning of the aggregate score | Report per-category and per-release scores |
| Public release delay | Latest private/delayed questions differ from public replication set | Distinguish leaderboard set from reproducibility set |
This is the deepest open question for LiveBench: can a public rolling benchmark preserve trust while changing often enough to remain useful? The answer depends less on model progress than on governance.
11. Broader implications for benchmark lifecycle design
LiveBench suggests that future Benchmark Lifecycle Design should adopt several norms.
First, every benchmark item should have a provenance record: source, generation method, release date, public-disclosure date, and retirement status. Without this, cutoff-aware evaluation becomes impossible.
Second, aggregate scores should be versioned. A model’s “LiveBench score” is incomplete without the release, task set, prompt format, scorer version, and evaluation harness. The same principle applies to any rolling benchmark.
Third, ambiguity and saturation should be treated as maintenance events. LiveBench’s removal of ambiguous or too-easy tasks is a feature, not a scandal, provided the change log is explicit. Static benchmarks often hide this problem by never updating.
Fourth, objective scoring should be used where possible, but not mistaken for complete evaluation. Verifiable benchmarks are strongest for math, code, structured reasoning, formatting, data manipulation, and constrained language tasks. They are weaker for judgment-heavy work.
Fifth, benchmark designers should expect adversarial optimization. Once a benchmark matters, models, prompts, scaffolds, and training loops will adapt to it. A good benchmark lifecycle assumes this and builds in refresh, task diversity, and provenance.
12. Bottom line
LiveBench’s importance is not just that it is another hard benchmark. Its importance is that it makes the benchmark itself temporal. Questions are not timeless probes of model ability; they are artifacts with publication dates, contamination risk, scorer assumptions, and eventual expiration.
The strongest practitioner lesson is to stop asking whether a benchmark is “the” benchmark. LiveBench, LiveCodeBench, MMLU-Pro, SWE-bench-style agentic tasks, human preference arenas, internal evals, and safety tests answer different questions. LiveBench’s contribution is to show how a broad public benchmark can remain alive after the first leaderboard cycle.
Selected sources
White et al. / LiveBench paper and official artifacts. The core sources are the arXiv/OpenReview LiveBench paper, the public GitHub repository, and the changelog. They establish the benchmark’s contamination-limited framing, objective scoring, six-category task design, original empirical results, and release history. GitHub+3arXiv+3OpenReview+3
LiveBench downstream implementations and current leaderboard context. Inspect Evals documents LiveBench release-date options, six-month refresh behavior, delayed latest release, scoring notes, and maintenance hazards; current official and third-party leaderboard snippets show the present model-score range and the risk of provenance fragmentation. LLM Stats+3UK Government BEIS+3LiveBench+3
LiveCodeBench. The LiveCodeBench paper and current leaderboard pages establish the code-only time-gated comparison point, including contest-problem sourcing, release-date-aware evaluation, code-generation and repair scenarios, and recent high-scoring frontier results. arXiv
MMLU and MMLU-Pro. The MMLU paper provides the static broad-domain baseline; MMLU-Pro provides the harder static successor with more answer options, more reasoning-heavy questions, and 14-domain coverage. arXiv+2arXiv+2
Recent leaderboard-analysis context. Vellum and Artificial Analysis illustrate the broader shift toward non-saturated, multi-benchmark, methodology-heavy model comparison rather than simple MMLU-style leaderboards. Vellum
Companion entries
Core theory: Benchmark Contamination, Time-Gated Evaluation, Benchmark Lifecycle Design, Evaluation Drift, Objective Scoring, LLM-as-Judge Bias
Benchmarks: LiveCodeBench, MMLU, MMLU-Pro, SWE-bench, IFEval, Big-Bench Hard, Humanity’s Last Exam
Practice: Evaluation Stack, Model Selection, Regression Evaluation, Release-Versioned Benchmarks, Benchmark Harnesses, Pass@1 Evaluation
Counterarguments: Leaderboard Overfitting, Benchmark Saturation, Private vs Public Evaluation, Verifiable Task Narrowness, Aggregate Score Misuse