1. What HumanEval is
HumanEval is the 164-problem Python code-generation benchmark introduced by Chen et al. in the Codex paper, Evaluating Large Language Models Trained on Code. The benchmark asks a model to synthesize Python function bodies from a function signature and natural-language docstring, then evaluates the generated code by executing unit tests. Chen et al. describe the dataset as “164 hand-written programming problems”; each problem includes a function signature, docstring, body, and several unit tests, with an average of 7.7 tests per problem. The authors emphasize that the tasks were hand-written because Codex had been trained on a large fraction of GitHub, where many existing programming-problem solutions already appeared. ar5iv
HumanEval’s design was narrow but decisive. It did not try to evaluate repository navigation, architectural judgment, debugging, testing strategy, API design, or software maintenance. It evaluated a simpler question: given a concise English specification embedded in a Python docstring, can the model produce a single correct Python function? That made the benchmark small enough to run often, deterministic enough to compare across systems, and semantically grounded enough to beat surface metrics such as BLEU. Chen et al. explicitly argued that match-based metrics fail for code because many programs can be functionally equivalent while textually different; they therefore centered execution-based Functional Correctness and Pass@k evaluation. ar5iv
The canonical HumanEval workflow is:
| Stage | What happens | Why it mattered |
|---|---|---|
| Prompt construction | The model receives a Python function signature plus an English docstring. | Turns code synthesis into a natural-language-to-code task. |
| Generation | The model emits a completion, usually the missing function body. | Allows direct comparison between model families. |
| Execution | The completion is run against hidden unit tests. | Measures behavior rather than textual similarity. |
| Aggregation | Results are reported as pass@k, usually pass@1, pass@10, or pass@100. | Captures both single-shot reliability and sampling-based search. |
The original OpenAI repository describes HumanEval as an evaluation harness for the problem-solving dataset from the Codex paper, and it warns users that the harness executes untrusted model-generated code and should not be run outside a robust security sandbox. GitHub This warning is not incidental. HumanEval helped normalize a now-standard pattern in code-model evaluation: generate code, isolate it, execute it, and grade behavior. That pattern is much closer to how programmers actually validate code than string matching, but it also imports all the brittleness of tests, sandboxes, dependency environments, timeouts, and prompt protocols.
2. The original design: why 164 small Python functions were enough
HumanEval’s most important design choice was not its size. It was the decision to evaluate program behavior. The Codex paper contrasts execution-based evaluation with match-based metrics and notes that human developers often judge code by whether it passes tests. The paper also gives an unbiased estimator for pass@k rather than naïvely extrapolating from empirical pass@1, because repeated sampling can otherwise produce biased estimates. ar5iv
The benchmark contains mostly compact tasks involving language comprehension, reasoning, algorithms, and simple mathematics. Chen et al. explicitly frame the dataset as a measure of problem-solving capability, not broad software-engineering competence. ar5iv That distinction matters. HumanEval was never a benchmark for “being a software engineer.” It was a benchmark for a particular act of synthesis: implement a small, self-contained Python function from a docstring.
This design had several strengths.
| Design feature | Strength | Hidden cost |
|---|---|---|
| Hand-written problems | Reduced pre-2021 leakage from common GitHub coding-problem repositories. | Post-release contamination became unavoidable once the benchmark was public. |
| Python-only | Simplified execution and made evaluation cheap. | Conflated “coding ability” with Python-docstring familiarity. |
| Function-level tasks | Made outputs short, executable, and easy to test. | Excluded state, multi-file coordination, architecture, APIs, and maintenance. |
| Unit-test grading | Measured behavior rather than textual similarity. | Rewarded passing a finite test suite, not proving semantic correctness. |
| pass@k | Captured stochastic generation and sampling. | Could make models look better when allowed many attempts. |
| Small dataset | Enabled rapid iteration. | Made leaderboard differences near saturation statistically fragile. |
HumanEval’s compactness is easy to underestimate. A 164-task benchmark means one task is about 0.61 percentage points. At a reported score of 96%, only about six or seven tasks remain unsolved; at 97.6%, about four tasks remain unsolved. Near the ceiling, differences of one or two percentage points can correspond to only a handful of problems, and those differences may be driven by prompt formatting, decoding details, harness differences, or benchmark exposure rather than durable capability differences.
3. Historical importance: the benchmark that made code LLMs legible
HumanEval was not the first program-synthesis benchmark in the broad history of AI or programming languages. But it was the first widely adopted LLM-era code-generation benchmark. Its importance came from timing, simplicity, and a good evaluation target.
The Codex paper reported that a 12B-parameter Codex model solved 28.8% of HumanEval with a single sample, while GPT-3 solved 0% and GPT-J solved 11.4%. With 100 samples per problem, Codex solved 70.2%. arXiv In the paper’s ar5iv rendering, the introduction similarly states that 12B Codex solved 28.8% with one sample, 300M Codex solved 13.2%, GPT-J reached 11.4%, and GPT models were near zero. ar5iv
Those numbers mattered because they showed three things at once.
First, code-specific training mattered. General language models had learned some syntax and snippets, but Codex’s GitHub fine-tuning produced qualitatively different Python synthesis behavior.
Second, scale mattered. The Codex family showed improved HumanEval results with larger models, making the benchmark part of the early Code LLM Scaling story.
Third, sampling mattered. The jump from pass@1 to pass@100 made repeated generation look like a primitive search procedure: if the model could produce at least one correct candidate in a batch, unit tests could select it. That result foreshadowed later agentic coding loops, self-repair, test generation, and execution-guided decoding.
HumanEval’s adoption was also helped by its low operational cost. A lab could evaluate a model without constructing a huge infrastructure benchmark. A company could add a number to a model card. Open-source projects could run it on consumer-accessible hardware for smaller models. Academic papers could compare against Codex, CodeGen, StarCoder, CodeLlama, WizardCoder, DeepSeek-Coder, Qwen-Coder, and later frontier chat models. The benchmark became a shared coordinate system for Code Generation Benchmarks.
4. The empirical trajectory: from Codex to saturation
The trajectory of HumanEval scores is a compressed history of neural code generation. The early benchmark separated code-specialized models from general LMs. By the mid-2020s, the frontier had largely saturated the base benchmark.
| Period | Representative result | What the result showed | Citation |
|---|---|---|---|
| 2021 Codex era | Codex 12B: 28.8% pass@1; 70.2% pass@100 | Code fine-tuning plus sampling made executable synthesis work. | arXiv |
| 2021 general-LM comparison | GPT-3: 0%; GPT-J: 11.4% | HumanEval exposed a sharp gap between general language modeling and code-trained synthesis. | arXiv |
| 2023–2024 test-strengthening era | HumanEval+ reduced apparent pass@k by up to 19.3–28.9% in EvalPlus experiments | Many “correct” HumanEval solutions were brittle under stronger tests. | arXiv |
| 2024 reasoning-model era | o1-mini reported at 96.2% pass@1 on HumanEval but 76.2% on HumanEval Pro | Traditional HumanEval was no longer discriminative for top models. | arXiv |
| 2026 public leaderboard era | Public aggregators report top HumanEval scores in the mid-90s to high-90s; one leaderboard lists Claude Sonnet 4.5 Thinking at 97.6%, R1 at 97.4%, and Grok 4 at 97.0% | The base benchmark has become a ceiling-region sanity check, not a frontier-ranking instrument. | Price Per Token |
The exact frontier number should be treated carefully. Public HumanEval leaderboards disagree because they differ in model set, prompt protocol, date, decoding settings, and whether they ingest vendor-reported or independently run results. For example, one 2026 public leaderboard lists a 97.6% top score, while another lists a 95.1% top score across its evaluated models. Price Per Token The robust conclusion is not that a particular frontier model is “the” HumanEval champion. The robust conclusion is that base HumanEval is saturated for frontier systems.
The saturation point is visible even in successor papers. HumanEval Pro reports that “most LLMs excel” on traditional HumanEval and MBPP, then shows a sharp drop on self-invoking code generation, where a model must solve a base problem and use that solution to solve a related harder problem. The paper’s example—o1-mini at 96.2% on HumanEval but 76.2% on HumanEval Pro—captures the problem: HumanEval still verifies small-function synthesis, but it no longer separates advanced reasoning and composition ability at the frontier. arXiv
5. What HumanEval measures, precisely
At its best, HumanEval measures whether a model can map a short English specification to a working Python function under a known harness. It is strongest as a test of local synthesis.
It measures:
| Capability | HumanEval signal |
|---|---|
| Python syntax and idiom | Can the model emit executable Python without scaffolding? |
| Local algorithmic reasoning | Can it infer loops, conditions, recursion, sorting, parsing, and arithmetic from a docstring? |
| Specification following | Can it preserve edge cases described in the prompt? |
| Single-function decomposition | Can it turn a small task into coherent implementation steps? |
| Basic semantic correctness | Does the implementation pass a finite test suite? |
It does not measure, or only weakly measures:
| Missing capability | Why HumanEval under-measures it |
|---|---|
| Repository-level change | No multi-file codebase, no imports across project boundaries, no version constraints. |
| Debugging over time | The canonical task is one-shot generation, not iterative diagnosis. |
| Requirement negotiation | The docstrings are compact and artificially settled. |
| Test design | The model is not asked to write a robust test suite. |
| Security | Passing tests does not imply safe, adversarially robust, or non-malicious code. |
| Maintainability | No review of readability, modularity, complexity, or long-term design. |
| Performance engineering | Most tasks are too small to expose asymptotic or systems-level tradeoffs. |
| Multilingual natural-language understanding | Original prompts are English. |
| API/tool use | Tasks are mostly self-contained rather than library-compositional. |
This is why HumanEval’s current role should be understood as a unit benchmark, not a comprehensive coding benchmark. A model that fails HumanEval badly is probably weak at Python synthesis. A model that scores 97% has not thereby demonstrated production software engineering competence.
6. Saturation and the ceiling problem
Benchmark Saturation occurs when a benchmark’s top scores cluster so close to the maximum that the benchmark loses discriminative power. HumanEval is a textbook case.
The ceiling problem has four layers.
Small n. HumanEval has 164 tasks. Around the frontier, one problem changes the score by about 0.61 percentage points. This makes fine-grained ranking unstable.
Task simplicity. The original problems are concise, function-level, and self-contained. That was the point in 2021. It is the weakness in 2026.
Public exposure. The benchmark has been public for years, appears in repositories, model-evaluation suites, blog posts, tutorials, and synthetic data pipelines, and is often directly included in training-data filtering logic.
Protocol drift. HumanEval numbers are reported under different prompts, decoding temperatures, chat wrappers, stop sequences, pass@k values, and execution harnesses. A score without protocol details is not a scientific result.
Saturation does not make HumanEval useless. It changes what the score means. In 2021, 28.8% pass@1 from Codex was a breakthrough. In 2026, a 96–98% score mostly says: this model can solve almost all of a small, public, English, single-function Python benchmark under the reported harness. That is still useful as a regression test. It is not enough for frontier claims.
7. Contamination: from design concern to central interpretive problem
The original HumanEval designers were already worried about contamination. Chen et al. made the tasks hand-written because Codex was trained on a large fraction of GitHub, and GitHub already contained solutions to many programming-problem sources. ar5iv That design choice reduced the chance that Codex had seen the exact tasks before July 2021. It did not protect the benchmark after release.
Once HumanEval became public, it entered the data ecosystem: GitHub forks, benchmark harnesses, blog tutorials, model-evaluation repos, prompt examples, leaderboard submissions, synthetic instruction datasets, and decontamination test sets. For any model trained after HumanEval’s publication, a high score may reflect some mixture of generalizable code capability, prompt familiarity, memorized solutions, memorized tests, memorized variants, and benchmark-specific fine-tuning.
The contamination literature makes this concern concrete.
Yang et al., in Rethinking Benchmark and Contamination for Language Models with Rephrased Samples, argue that string matching is insufficient because paraphrases and translations can bypass decontamination. They validate the issue on benchmarks including HumanEval and report that, in pretraining sets such as RedPajama-Data-1T and StarCoder-Data, they identified 8–18% HumanEval overlap. They also warn that synthetic datasets generated by GPT-3.5/4 can unintentionally contain benchmark contamination. arXiv
Dong et al., in Generalization or Memorization, focus on detecting contamination through output-distribution behavior. They argue that open-access benchmarks may be included in training data, intentionally or unintentionally, and report that ChatGPT showed high potential for contamination on HumanEval. arXiv
Riddell, Ni, and Cohan specifically study code-generation contamination. Their paper, Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models, uses surface-level and semantic-level matching to quantify overlap between popular code-generation benchmarks and pretraining corpora. They report substantial overlap and find that models perform significantly better on benchmark subsets where similar solutions were seen during training. arXiv
Golchin and Surdeanu’s Data Contamination Quiz work is relevant because it shows how black-box contamination detection can be framed as a multiple-choice task: if a model reliably selects the exact original instance over perturbed alternatives, that can indicate exposure. Their method does not require access to training data, parameters, or logits. arXiv
The contamination issue is not merely “the model saw the test.” In code benchmarks, contamination can take several forms.
| Contamination mode | Example | Why it matters |
|---|---|---|
| Prompt contamination | The exact HumanEval docstring appears in training. | The model may learn the problem statement directly. |
| Solution contamination | Canonical or community solutions appear in training. | The model can reproduce or adapt memorized code. |
| Test contamination | Unit tests appear in training. | The model can overfit to the grader. |
| Semantic contamination | Reworded versions or near-equivalent tasks appear. | String decontamination may miss it. |
| Synthetic contamination | Instruction-tuning data generated from models that know HumanEval includes variants. | Leakage can propagate indirectly. |
| Evaluation-harness contamination | Training data includes prompt templates and expected completion format. | The model may learn benchmark-specific conventions. |
This does not imply that every high HumanEval score is fake. It means HumanEval cannot be interpreted as a clean generalization test for post-2021 models unless the evaluator has strong evidence about training data, filtering, and exposure. Closed frontier models rarely provide enough evidence for that.
8. Unit-test brittleness and HumanEval+
HumanEval’s other major weakness is not contamination but test insufficiency. A finite unit test suite can certify that a program passes those tests, not that it implements the full intended function. HumanEval averaged 7.7 tests per problem; many incorrect programs can pass such a suite by mishandling untested edge cases.
EvalPlus and HumanEval+ were built to address this. Liu et al.’s EvalPlus paper argues that existing benchmark tests are limited in quantity and quality, then augments HumanEval with large numbers of automatically generated tests. The authors report that HumanEval+ extends HumanEval tests by 80× and catches significant amounts of previously undetected wrong code, reducing pass@k by up to 19.3–28.9%. They also found that test insufficiency can mis-rank models. arXiv
The EvalPlus repository summarizes the operational result: HumanEval+ has 80× more tests than the original HumanEval, and MBPP+ has 35× more tests than original MBPP. GitHub The EvalPlus leaderboard also cautions that providers bear responsibility for avoiding contamination and that models trained on closed data can be affected by contamination. evalplus.github.io
This critique is subtle. HumanEval’s original tests were not “bad” for 2021. They were good enough to reveal a new phenomenon: LLMs could synthesize executable Python. But as models improved, weak tests became a ceiling amplifier. A model could pass because it generalized correctly, because it guessed common cases, because it had seen the task, or because the tests failed to expose a wrong edge case. HumanEval+ tightened one part of that uncertainty: functional brittleness.
9. Methodology critiques
9.1 Function-only scope
HumanEval is a function-level benchmark. That scope makes it reproducible and cheap, but it leaves out much of programming. Real software engineering often involves understanding existing abstractions, preserving invariants, integrating with external libraries, reading errors, editing multiple files, handling version constraints, and balancing competing requirements. HumanEval contains none of that.
ClassEval’s authors make this critique directly. They argue that standalone method-level benchmarks like HumanEval do not equivalently reflect class-level coding ability. Their ClassEval benchmark contains 100 class-level Python tasks, and they find that existing LLMs perform much worse on class-level code generation than on standalone method-level benchmarks. arXiv The ClassEval repository further describes 100 classes, 410 methods, and an average of 33.1 test cases per class, with field, method, library, and standalone dependencies. GitHub
9.2 English-language docstrings
Original HumanEval is English-to-Python. This is reasonable for a first benchmark, but it is not a neutral measure of multilingual programming assistance. HumanEval-XL’s authors explicitly note that existing benchmarks mainly concentrated on English prompts to multilingual code or used very limited natural languages; they introduce 23 natural languages and 12 programming languages to test cross-lingual natural-language generalization. arXiv
This matters because many code assistants are used globally. A model can score well on English docstrings and still perform poorly when requirements are written in Japanese, Arabic, Spanish, Hindi, or mixed natural-language/code idioms. HumanEval does not reveal that gap.
9.3 Unit-test brittleness
Passing HumanEval means passing the included tests. It does not mean full semantic correctness. EvalPlus demonstrated that adding stronger tests can substantially reduce apparent scores and alter model rankings. arXiv
9.4 Prompt and harness sensitivity
HumanEval was originally designed for completion-style models. Modern frontier systems are often chat models, reasoning models, or tool-using agents. Wrapping a docstring benchmark in a chat prompt changes the task. Some models may include explanations unless instructed not to; some may generate full files instead of completions; some may rely on hidden reasoning; some may perform better with self-debugging or tool access. HumanEval scores are only comparable when prompt, decoding, stop conditions, execution environment, and pass@k are comparable.
9.5 pass@k as both metric and search budget
pass@k is mathematically appropriate for stochastic generation, but it can obscure the distinction between “the model reliably writes correct code” and “the model sometimes writes correct code if sampled enough times.” The Codex paper’s pass@100 result was scientifically important because it revealed a search-like capability. arXiv But for product claims, pass@1, pass@k with cost disclosure, and agentic repair success are different measures.
9.6 Small benchmark, high reuse
The more a benchmark is reused, the less it behaves like an unseen test. HumanEval’s convenience made it ubiquitous; ubiquity made it less clean. That is the standard benchmark lifecycle: adoption produces comparability, comparability produces optimization, optimization produces saturation and contamination.
10. Successors and complements
HumanEval’s successors are best understood as responses to specific failure modes, not as a single replacement.
| Benchmark | Introduced to address | Unit of work | Main contribution | Relevant source |
|---|---|---|---|---|
| HumanEval+ / EvalPlus | Weak unit tests | Same HumanEval tasks with stronger tests | Adds large generated test suites; catches brittle wrong solutions; reduces inflated pass@k. | arXiv |
| MBPP | Need for more short Python tasks | Short Python programs | 974 crowd-sourced tasks solvable by entry-level programmers; natural-language descriptions plus tests. | arXiv |
| MBPP+ | Weak MBPP tests | MBPP subset with stronger tests | EvalPlus extension with 35× more tests than original MBPP. | GitHub |
| LiveCodeBench | Contamination and static benchmark saturation | Contest-style coding tasks over time | Continuously collects new problems from LeetCode, AtCoder, and Codeforces; includes self-repair, code execution, and test-output prediction. | arXiv |
| BigCodeBench | Function-only simplicity and lack of library/tool composition | Practical Python tasks with library calls | 1,140 tasks using 139 libraries across 7 domains, with complex instructions and high branch coverage. | arXiv |
| SWE-bench | Lack of repository-level software engineering | Real GitHub issue resolution | 2,294 tasks from issues and PRs across 12 Python repositories; models must edit codebases to resolve issues. | arXiv |
| SWE-bench Verified | Need for a cleaner, human-filtered SWE-bench subset | Repository-level issue resolution | 500 human-filtered instances, commonly used for frontier coding-agent comparisons. | swebench.com |
| ClassEval | Lack of class-level compositional generation | Multi-method Python classes | 100 class-level tasks with method, field, and library dependencies. | arXiv |
| HumanEval-XL | English-only natural-language prompts | Multilingual NL-to-code | 23 natural languages, 12 programming languages, 22,080 prompts. | arXiv |
MBPP
MBPP—Mostly Basic Programming Problems—was introduced by Austin et al. as part of Program Synthesis with Large Language Models. It contains 974 short Python programming tasks designed to be solvable by entry-level programmers, and the paper evaluates models from 244M to 137B parameters. The authors report that synthesis performance scales log-linearly with model size, and that their largest models could synthesize solutions to 59.6% of MBPP problems in few-shot prompting without code-dataset fine-tuning. arXiv
MBPP is closest to HumanEval in spirit: short Python synthesis from natural language. Its value is breadth. Its weakness is similar: small task granularity, Python focus, and finite tests.
HumanEval+
HumanEval+ is the direct repair patch for HumanEval’s test brittleness. It keeps the original tasks but strengthens the grader. It is best used when one wants historical continuity with HumanEval but does not want to trust the original test suites.
LiveCodeBench
LiveCodeBench is the most direct response to static contamination. It continuously collects new contest problems over time from LeetCode, AtCoder, and Codeforces, annotates release dates, and allows time-sliced evaluation after a model’s training cutoff. Its authors explicitly state that existing benchmarks such as HumanEval and MBPP are no longer sufficient because of contamination, overfitting, saturation, and narrow focus. arXiv
LiveCodeBench also broadens the task mix beyond generation to self-repair, code execution, and test-output prediction. That makes it closer to a Dynamic Benchmark than a fixed test set.
BigCodeBench
BigCodeBench targets a different weakness: HumanEval’s lack of real library composition. Its authors argue that current benchmarks are often limited to short, self-contained algorithmic tasks or standalone function calls. BigCodeBench asks models to solve 1,140 fine-grained tasks using multiple function calls from 139 libraries across 7 domains, with 5.6 tests per task and average branch coverage of 99%. arXiv
The key shift is from “write a small function” to “understand an instruction and use available APIs precisely.” That is closer to many practical automation tasks.
SWE-bench
SWE-bench is a category shift rather than an incremental successor. It asks whether models can resolve real GitHub issues by editing codebases. The original benchmark contains 2,294 software-engineering problems drawn from GitHub issues and corresponding pull requests across 12 Python repositories. The model receives a codebase and issue description and must generate a patch. arXiv
SWE-bench’s authors emphasize that resolving issues often requires changes across multiple functions, classes, and files, plus interaction with execution environments and long contexts. arXiv That is precisely what HumanEval excludes. SWE-bench is therefore a better benchmark for coding agents, though it introduces its own complexities: scaffolds, tools, cost budgets, repository setup, flaky tests, and patch validation.
ClassEval
ClassEval sits between HumanEval and SWE-bench. It does not require full repository issue resolution, but it does require generating classes with multiple methods and dependencies. It tests compositional structure, shared state, field dependencies, method dependencies, and library dependencies. The authors find that class-level performance is much worse than standalone method-level performance, which directly challenges the inference “high HumanEval implies broad code-generation ability.” arXiv
11. What HumanEval still measures at saturation
HumanEval still measures something. The mistake is to confuse that something with the whole field.
At saturation, HumanEval is useful for:
| Use | Why it remains useful |
|---|---|
| Smoke testing | A serious Python code model should not fail badly on HumanEval. |
| Regression testing | A model update that drops sharply on HumanEval probably broke basic synthesis. |
| Historical comparison | HumanEval gives continuity from Codex to modern systems. |
| Harness validation | It is a convenient way to test execution infrastructure, pass@k logic, and sandboxing. |
| Small-model comparison | For non-frontier or highly compressed models, it may still have spread. |
| Teaching benchmark methodology | It is simple enough to explain execution-based evaluation. |
HumanEval should be retired from:
| Use | Why it should be retired |
|---|---|
| Frontier model ranking | Scores are too close to the ceiling and too protocol-sensitive. |
| Claims of software-engineering competence | The benchmark is function-only and repository-free. |
| Contamination-sensitive evaluation | The benchmark is public and widely replicated. |
| Product-level coding claims | It does not measure debugging, maintenance, tools, or user interaction. |
| Safety claims | Passing tests does not imply secure or non-harmful code behavior. |
| Agent evaluation | It lacks multi-step interaction, environment use, and patch validation. |
A reasonable rule is: report HumanEval only as a legacy baseline, never as the headline result for a frontier coding system. Pair it with HumanEval+, MBPP+, LiveCodeBench, BigCodeBench, SWE-bench Verified, and task-specific internal evaluations.
12. The open question: capability, contamination, or convention?
The open interpretive question is what a 96–98% HumanEval score means in 2026.
There are three competing interpretations.
Capability interpretation: the model has learned robust small-function Python synthesis and can solve nearly all HumanEval tasks from the specification alone.
Contamination interpretation: the model has seen enough of the benchmark, its solutions, its tests, or close variants that the score partly reflects memorization.
Convention interpretation: the model has learned the style of HumanEval-like prompts and the standard benchmark completion protocol, which helps even without exact memorization.
All three can be true simultaneously. The hard part is disentangling them. For closed models, the evaluator usually lacks training-data access. For open models, the data pipeline may still be too large for perfect auditing. For synthetic data, provenance can be recursive: a model trained on synthetic coding examples may inherit benchmark knowledge from the generator.
This is why dynamic and time-sliced benchmarks matter. LiveCodeBench’s release-date structure is one answer: evaluate on problems published after the model’s training cutoff. livecodebench.github.io Private one-time exams are another answer, though they sacrifice public reproducibility. Stronger decontamination methods are a third answer, but Yang et al.’s paraphrase result shows why simple string overlap is inadequate. arXiv
The deeper issue is epistemic. Public benchmarks are coordination devices. They let a field compare progress. But the more successful a public benchmark becomes, the more it becomes part of the training distribution, the evaluation culture, and the optimization target. HumanEval is not a failed benchmark. It is a benchmark that succeeded so completely that it exhausted its original role.
13. Practical reporting standard for HumanEval
A technically honest HumanEval report should include:
| Required detail | Reason |
|---|---|
| Exact benchmark version | HumanEval variants and harnesses differ. |
| Prompt template | Chat and completion prompts are not equivalent. |
| Decoding parameters | Temperature, top-p, number of samples, and stop sequences affect pass@k. |
| pass@k definition | pass@1 and pass@100 measure different things. |
| Execution environment | Python version, sandbox, timeout, dependencies, and disabled imports matter. |
| Whether HumanEval+ was also run | Base tests are weak. |
| Training cutoff and decontamination statement | Public benchmark exposure is central. |
| Whether results are independent or vendor-reported | Leaderboard provenance matters. |
| Confidence or task-count interpretation | Near-ceiling differences are small in absolute task count. |
For frontier systems, HumanEval should be reported in a bundle:
| Evaluation layer | Recommended benchmark family |
|---|---|
| Legacy small-function synthesis | HumanEval, HumanEval+ |
| Broader small Python synthesis | MBPP, MBPP+ |
| Fresh contest-style coding | LiveCodeBench |
| Library/API composition | BigCodeBench |
| Class-level composition | ClassEval |
| Repository-level issue resolution | SWE-bench Verified / SWE-bench Full |
| Internal product fit | Private, task-specific evals with fresh tasks |
14. Bottom line
HumanEval’s historical contribution was to make code-generation evaluation executable, comparable, and simple. It helped show that code-trained LLMs were not merely producing plausible snippets but could synthesize functions that passed tests. The Codex result—28.8% pass@1 and 70.2% pass@100—was a genuine signal of a new capability. arXiv
Its current status is different. HumanEval is a saturated, public, function-level benchmark with known contamination and test-strength concerns. EvalPlus, LiveCodeBench, BigCodeBench, SWE-bench, and ClassEval exist because HumanEval does not measure enough of what modern code models and coding agents claim to do. arXiv+4arXiv+4arXiv+4
The right retirement plan is not deletion. HumanEval should remain as a historical baseline and regression test. It should be retired as a frontier headline number.
Companion entries
Core theory: Functional Correctness, Pass@k, Program Synthesis, Code LLM Scaling, Benchmark Saturation, Execution-Based Evaluation
Benchmark lineage: HumanEval+, EvalPlus, MBPP, LiveCodeBench, BigCodeBench, SWE-bench, SWE-bench Verified, ClassEval, HumanEval-XL
Methodology: Benchmark Contamination, Data Contamination Quiz, Unit-Test Adequacy, Dynamic Benchmarks, Private Evaluation Sets, Prompt Protocols for Code Evaluation
Practice: Coding Agent Evaluation, Repository-Level Software Engineering Benchmarks, Sandboxing Generated Code, Test-Driven Development, Model Card Benchmark Reporting
Counterarguments: Why Legacy Benchmarks Still Matter, The Case Against Leaderboards, Benchmark Overfitting, Contamination vs Generalization, When pass@1 Is Not Enough