MMLU-Redux is an item-quality audit of MMLU that treats benchmark maintenance as a measurement problem rather than a leaderboard afterthought. Its central claim is narrow but consequential: a nontrivial fraction of original MMLU questions are mislabeled, ambiguous, underspecified, or otherwise unsuitable for single-answer multiple-choice evaluation, and those defects can change model-level conclusions in specific subjects. arXiv
Coverage note: evidence checked through May 18, 2026, against the MMLU-Redux NAACL paper, public MMLU-Redux dataset materials, MMLU-Pro, MMLU-CF, and adjacent benchmark-audit literature.
Why MMLU-Redux exists
MMLU became one of the canonical general-knowledge and reasoning benchmarks for large language models because it spans 57 subjects, from elementary mathematics and U.S. history to law, medicine, computer science, and professional exams. The original MMLU paper framed the benchmark as a broad test of world knowledge and problem solving, explicitly designed to evaluate both breadth and depth across many academic and professional domains. arXiv
That breadth is also the source of MMLU’s maintenance problem. A 14,000-plus item pool assembled from many domains is not a static truth object; it is a large collection of exam-like artifacts whose labels, wording, options, and background assumptions can be wrong. MMLU-Redux asks whether high-stakes model comparisons should continue treating every MMLU item as equally valid, or whether the benchmark needs Dataset Auditing and versioned correction like any other engineered measurement instrument. arXiv
The MMLU-Redux authors report that they manually re-annotated 5,700 MMLU questions, sampling 100 questions from each of the 57 subjects, with 14 human experts involved in assessment and re-annotation. Their headline estimate is that 6.49% of the full MMLU test set contains correctness or ambiguity issues, with much higher defect rates in some subjects, especially Virology. arXiv
The project therefore belongs to a broader family of benchmark-quality audits: ImageNet label audits, natural-language-inference artifact studies, label-error audits across popular test sets, “Fantastic Bugs” style benchmark invalidity detection, and recent audits of text-to-SQL benchmarks. The common lesson is that benchmark scores are not only properties of models; they are also properties of the measurement instrument. vldb.org+3arXiv+3ACL Anthology+3
What MMLU-Redux is — and is not
MMLU-Redux is best understood as a curated error annotation layer over the original MMLU item pool, not as a wholly new benchmark in the style of MMLU-Pro or MMLU-CF. It preserves the original MMLU question text, answer choices, and original answer labels, then adds item-level metadata: whether the question is acceptable, what kind of problem it has if not, source evidence where available, a possible corrected answer where applicable, and notes about the likely source of the defect. arXiv
That distinction matters. MMLU-Redux does not claim to have rewritten every problematic item into a new polished benchmark. It provides a diagnosis and partial repair surface: evaluators can filter out bad items, correct wrong answer keys where the intended answer is clear, or study how model rankings change when suspect items are removed. The dataset card explicitly says the original question, choices, and answer columns remain the same as MMLU, while additional columns such as error_type, source, correct_answer, and potential_reason are used for auditing and filtering. arXiv
A useful mental model is:
| Layer | What it contains | What MMLU-Redux changes |
|---|---|---|
| Original MMLU item | Question, choices, original answer key | Preserved |
| Audit judgment | Whether the item is valid | Added |
| Error taxonomy | Kind of defect, if any | Added |
| Correction hint | Correct answer where inferable | Added for relevant cases |
| Benchmark policy | Whether to keep, fix, remove, or rewrite | Left to downstream users and maintainers |
The article’s subtitle, “Cleaning Up the Item Pool,” should therefore be read carefully. MMLU-Redux enables cleaning; it is not itself a single universally adopted clean MMLU replacement.
Construction methodology
Sampling design
MMLU-Redux sampled 100 questions from each of the 57 MMLU subjects, yielding 5,700 audited questions. This is not a proportional sample of the original test set, because MMLU subjects have different original sizes. The paper’s global 6.49% estimate is therefore computed as a stratified estimate over the full 14,042-item MMLU test split, not merely as the raw unweighted fraction of sampled questions. arXiv
The sampling choice has an important consequence for interpretation. It gives every subject a visible audit footprint, including small subjects such as Virology and College Chemistry, but it also means that subject-level defect rates should not be averaged naively if the goal is to estimate full-benchmark defect mass. The paper separately reports very high error rates in some sampled subjects and a lower weighted full-test estimate. arXiv
Expert annotation workflow
The annotation procedure combined source tracing and expert judgment. Annotators searched for exact matches to original exam or source material where possible; when a match was found, they compared the MMLU answer key against the source. Even when exact sources were unavailable, annotators used trusted references such as textbooks, government websites, and reputable organizations, and could mark cases as requiring more domain expertise. arXiv
The project’s methodology is deliberately closer to forensic benchmark auditing than to crowdsourced relabeling. It asks not just “what answer seems right?” but “is this a well-posed single-answer multiple-choice question under the intended domain assumptions?” That distinction is central for LLM evaluation, because an item can be answerable by an expert yet still invalid as a benchmark item if several choices are defensible, the intended jurisdiction is missing, the wording is underspecified, or the original answer key is wrong.
Error taxonomy
MMLU-Redux divides error types into two broad groups: question/option quality problems and ground-truth verification problems. The core labels are Bad Question Clarity, Bad Options Clarity, No Correct Answer, Multiple Correct Answers, and Wrong Ground Truth. arXiv
| MMLU-Redux label | Abbrev. | Meaning | Typical downstream treatment |
|---|---|---|---|
| Bad Question Clarity | BQC | The question is ambiguous, underspecified, misleading, or missing needed context. | Remove or rewrite; do not merely relabel. |
| Bad Options Clarity | BOC | The answer choices are unclear, malformed, overlapping, or otherwise unsuitable. | Remove or rewrite choices. |
| No Correct Answer | NCA | None of the listed options is correct. | Remove or add a valid option; sometimes supply correct_answer outside the original choices. |
| Multiple Correct Answers | MCA | More than one option is defensibly correct. | Remove, rewrite, or support multi-label scoring. |
| Wrong Ground Truth | WGT | The original answer key selects the wrong option. | Correct label if the intended item is otherwise valid. |
This taxonomy is useful because not all defects are equally repairable. A wrong answer key can often be patched cleanly. A badly underspecified question cannot be fixed by swapping labels; it requires rewriting the item or excluding it from single-label evaluation. A multiple-correct-answer item can be salvaged only if the scoring protocol changes or the distractors are rewritten.
Corrected and removed item lists
MMLU-Redux’s practical contribution is not just the aggregate defect estimate but the item-level audit table. The public dataset contains 57 subject subsets with 100 rows each, and its documented columns include the original MMLU fields plus error_type, source, correct_answer, and potential_reason. Hugging Face
That structure supports three downstream item lists:
| Derived list | How to construct it from MMLU-Redux | Evaluation meaning |
|---|---|---|
| Clean keep-list | Items marked acceptable / not erroneous | Use for “valid-item-only” scoring. |
| Correctable-label list | Mostly WGT items, and some NCA cases where the correct answer is clear | Use for corrected-key experiments, with disclosure. |
| Removal / rewrite list | BQC, BOC, MCA, many NCA cases | Exclude from single-answer scoring unless rewritten and revalidated. |
The paper notes that correct answers suggested by annotators were not simply used to replace original MMLU labels in the original benchmark. That is a conservative choice: it separates the empirical claim “this item is defective” from the governance claim “this new label should become the official benchmark.” arXiv
Empirical findings
The headline estimate: 6.49% defective items
Across the full MMLU test set, MMLU-Redux estimates that 93.51% of items are acceptable and 6.49% contain one of the audited error types. The estimated defect distribution is: 2.47% Bad Question Clarity, 0.44% Bad Options Clarity, 0.62% No Correct Answer, 1.54% Multiple Correct Answers, and 1.42% Wrong Ground Truth. arXiv
| Estimated full-test category | Share of original MMLU test set |
|---|---|
| Acceptable / OK | 93.51% |
| Bad Question Clarity | 2.47% |
| Bad Options Clarity | 0.44% |
| No Correct Answer | 0.62% |
| Multiple Correct Answers | 1.54% |
| Wrong Ground Truth | 1.42% |
| Any audited defect | 6.49% |
The distinction between “wrong answer key” and “bad item” is important. Only a subset of defective items are simple label errors. Many are validity failures: ambiguous wording, overlapping options, missing context, or multiple defensible answers. Those are harder to correct without changing the task itself.
Defects are concentrated by subject
The most striking result is not only the global defect rate but its subject skew. In the audited sample, Virology had 57 defective items out of 100, including 33 Wrong Ground Truth cases. Logical Fallacies had 26 defective items, College Chemistry 25, Professional Law 18, Human Sexuality 17, Business Ethics 14, Formal Logic 13, Human Aging 12, Global Facts 12, and Machine Learning 11. arXiv+2arXiv+2
| Subject | Original subject size | Audited items | OK | Any defect | Notable pattern |
|---|---|---|---|---|---|
| Virology | 166 | 100 | 43 | 57 | Many wrong ground-truth labels; substantial ambiguity. |
| Logical Fallacies | 163 | 100 | 74 | 26 | Clarity and label problems. |
| College Chemistry | 100 | 100 | 75 | 25 | Many wrong ground-truth cases. |
| Professional Law | 1,534 | 100 | 82 | 18 | Jurisdiction and context sensitivity; multiple-correct cases. |
| Human Sexuality | 131 | 100 | 83 | 17 | Clarity issues and some label defects. |
| Business Ethics | 100 | 100 | 86 | 14 | Mostly question clarity problems. |
| Formal Logic | 126 | 100 | 87 | 13 | Multiple-correct and label issues. |
| Global Facts | 100 | 100 | 88 | 12 | Source conflicts and factual instability. |
| Machine Learning | 112 | 100 | 89 | 11 | Some outdatedness, ambiguity, and option-quality problems. |
The high-error subjects are heterogeneous. Virology’s problem is heavily label-related. Professional Law is more context-sensitive: jurisdiction, missing facts, and legal assumptions can determine the answer. Global Facts can depend on source choice and temporal validity. Machine Learning includes cases where terminology or state-of-the-field drift can make an old exam-style item questionable. arXiv
This heterogeneity is a warning against a single “MMLU has X% noise” slogan. The practical effect of item defects depends on which subject slice is being used, which model family is being compared, whether models are near saturation, and whether the defect type rewards memorization, penalizes correct reasoning, or merely adds noise.
Annotation reliability
MMLU-Redux reports inter-annotator agreement for five high-error subjects, with Cohen’s kappa values generally above 0.6. Reported agreement included Virology at 0.67, Logical Fallacies around 0.71–0.73, College Chemistry around 0.92–0.95, Formal Logic around 0.96–1.0, and Human Sexuality at 0.64, depending on whether binary error detection or full error-type labels were measured. arXiv
Those values support the claim that the audit is not merely arbitrary preference labeling. They do not eliminate all uncertainty. The paper itself notes limitations, including the need to annotate the remaining 8,342 MMLU test questions and the possibility of annotator bias. arXiv
Automatic detection remains weak
The authors also explored whether LLMs could automatically detect erroneous MMLU items. Their experiments with models such as GPT-4 Turbo, GPT-4o, Claude 3 Opus, and Llama 3 70B found that few-shot chain-of-thought prompting improved detection, but even the best non-retrieval F2 score was modest; retrieval-augmented approaches increased recall but still did not make automatic repair reliable enough to replace expert audit. arXiv
This matters for Self-Improving Evaluation Systems. MMLU-Redux suggests that LLMs can help triage suspicious benchmark items, but the final authority still needs domain-aware human review, source verification, and versioned governance. In other words, model-assisted benchmark maintenance is plausible; model-only benchmark maintenance is not yet justified.
Implications for cross-model comparisons
Item defects can change rankings
The paper’s re-evaluation experiments show that removing or isolating erroneous items can alter rankings in high-defect subjects. One example reported by the authors is that Llama 3.1 Instruct Turbo 405B ranked 16th on all Virology instances but first when evaluated only on correct Virology instances. Another example is GPT-4 on Human Sexuality: the paper reports a strong all-item score and rank, but a much lower correct-only score and last-place rank among the compared top-10 models. arXiv
The key implication is not that every published MMLU leaderboard is invalid. It is that leaderboard conclusions become fragile when score differences are small, subject slices are noisy, or models differ in their tendency to reproduce contaminated or erroneous answer keys. A model can score well on a defective item by matching the benchmark’s wrong label, while a model that reasons correctly can be penalized.
Measurement error is not symmetric
A tempting response is: “If all models face the same bad items, comparisons remain fair.” That is only true under restrictive assumptions. It requires item defects to affect all models equally, and it requires errors to be independent of model capabilities, training data, memorization, instruction-following behavior, and refusal behavior.
Those assumptions are weak. A model trained on benchmark-adjacent material may reproduce an erroneous answer key. A stronger domain model may reject the flawed premise and choose a different answer. A model optimized for multiple-choice exams may learn to infer intended answers from flawed wording, while another model may be more literal. In such cases, defective items introduce structured measurement error, not merely random noise.
A simple decomposition is:
| Component | Interpretation |
|---|---|
| Valid-item accuracy | Performance on items whose wording and labels support the benchmark’s intended construct. |
| Wrong-key alignment | Probability the model matches an erroneous benchmark label. |
| Ambiguity handling | How the model behaves when multiple answers are defensible or no answer is correct. |
| Prompt/scoring artifact | Whether formatting, refusal, or choice-selection behavior interacts with flawed items. |
MMLU-Redux primarily attacks the second and third components. It does not solve every MMLU concern, but it makes visible a class of errors that ordinary aggregate accuracy hides.
Old MMLU comparisons need confidence bands, not erasure
The right response to MMLU-Redux is not to discard every historical MMLU result. Original MMLU remains a large, influential, broad-coverage benchmark. Many comparisons with large score gaps are probably directionally robust. But fine-grained claims — especially claims based on narrow subject slices, marginal leaderboard differences, or “model X beats model Y by 0.3 points” — should be treated with more caution.
For technical reporting, the safest practice is:
| Reporting choice | Why it matters |
|---|---|
| State the MMLU version and evaluation harness | Prompting and scoring protocols can change results. |
| Report original MMLU and cleaned / audited subsets separately | Preserves comparability while exposing validity sensitivity. |
| Avoid overinterpreting sub-percentage differences | Defect rates can exceed leaderboard gaps. |
| Break out high-error subjects | Aggregate scores can mask domain-specific failures. |
| Disclose whether wrong keys were corrected or removed | Correction and filtering answer different questions. |
| Include confidence intervals or bootstrap uncertainty where possible | Item-level noise is part of benchmark uncertainty. |
The deeper implication is for Leaderboard Hygiene. Benchmarks should be treated as versioned measurement systems. When the item pool changes, the benchmark version changes. When the scoring key changes, the benchmark version changes. When items are filtered, the benchmark version changes. Without this discipline, “MMLU score” becomes an ambiguous label covering multiple incompatible measurement procedures.
Relationship to MMLU-Pro
MMLU-Pro and MMLU-Redux respond to different weaknesses in MMLU. MMLU-Pro argues that original MMLU became too easy and too knowledge-driven for modern models, so it constructs a harder, more reasoning-focused benchmark with more answer choices and curated questions. It expands answer options from four to ten, integrates more challenging reasoning-oriented questions, removes trivial and noisy questions, and reports lower model accuracies plus reduced prompt sensitivity. arXiv
MMLU-Redux instead asks whether the original MMLU item pool is internally valid. Its intervention is not primarily “make the benchmark harder.” It is “identify which items should not be trusted as originally labeled.” That means the two projects are complementary, not substitutes.
| Dimension | MMLU-Redux | MMLU-Pro |
|---|---|---|
| Primary problem | Original MMLU contains erroneous, ambiguous, or invalid items. | Original MMLU is too easy, too knowledge-heavy, and insufficiently discriminative. |
| Main intervention | Audit original items; label error types; provide correction metadata. | Build a harder benchmark with more reasoning focus and more answer choices. |
| Relation to original MMLU | Diagnostic layer over MMLU. | New benchmark derived from and extending beyond MMLU. |
| Main validity question | “Are these items correctly posed and labeled?” | “Does this benchmark still separate strong models?” |
| Typical use | Cleaned scoring, item filtering, benchmark maintenance. | Harder leaderboard evaluation. |
MMLU-Pro’s remediation strategy is constructive: make a new, harder exam. MMLU-Redux’s strategy is curatorial: inspect the existing exam and mark defective questions. A mature evaluation ecosystem needs both. Harder invalid questions are still invalid; perfectly valid questions that all frontier models solve are no longer discriminative.
Relationship to MMLU-CF
MMLU-CF addresses a third problem: contamination. Because MMLU is public and widely used, its items can appear in training data, benchmark documentation, evaluation code, or model-development workflows. MMLU-CF proposes a contamination-resistant benchmark design, including broader-domain question sourcing, decontamination rules, a public validation set, and a closed test set. arXiv
The MMLU-CF paper and project materials frame contamination as both unintentional and malicious: models may accidentally train on benchmark items, or developers may overfit to public test questions. MMLU-CF reports that GPT-4o scores substantially lower on MMLU-CF than on original MMLU, and the ACL paper reports evaluation of more than 40 LLMs with performance drops and rank changes. arXiv
MMLU-Redux and MMLU-CF therefore target different failure modes:
| Failure mode | MMLU-Redux response | MMLU-CF response |
|---|---|---|
| Wrong labels | Audit and annotate item-level key errors. | Not the main focus, though validation review can catch some errors. |
| Ambiguous or invalid items | Taxonomize and flag problematic wording/options. | Cleaning pipeline and public validation, but contamination resistance is central. |
| Public-test contamination | Not the main focus. | Closed test set and decontamination rules. |
| Benchmark saturation | Indirectly relevant. | More challenging contamination-free evaluation. |
| Governance | Public dataset with community discussion and versioning. | Public validation plus closed test policy. |
In short: MMLU-Redux is about Benchmark Validity, MMLU-Pro is about Benchmark Difficulty, and MMLU-CF is about Benchmark Contamination. Real benchmark maintenance has to handle all three at once.
Broader item-quality-audit literature
MMLU-Redux is part of a larger pattern: once benchmarks become important enough to guide research and product decisions, their item pools receive forensic scrutiny, and that scrutiny often finds defects.
Label errors across popular test sets
Northcutt, Athalye, and Mueller’s “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks” estimated label errors across 10 commonly used computer-vision, NLP, and audio datasets. They estimated an average of at least 3.3% test-label errors and at least 6% in ImageNet validation, then showed that corrected labels can affect model selection, including cases where lower-capacity models appear preferable under corrected labels. arXiv
The analogy to MMLU-Redux is direct but incomplete. Northcutt et al. focus heavily on label errors, while MMLU-Redux’s taxonomy includes label errors plus invalid-question errors. For LLM multiple-choice benchmarks, ambiguity and construct validity are often as important as mislabeled answers.
Annotation artifacts in NLI
Gururangan et al.’s work on Annotation Artifacts in natural language inference showed that hypothesis-only models could predict labels surprisingly well on SNLI and MultiNLI, with artifacts such as negation and vagueness correlating with specific classes. The result was not merely “some labels are wrong,” but that dataset construction procedures can encode shortcuts that models exploit. ACL Anthology
MMLU-Redux is not primarily an artifact paper, but it shares the same measurement lesson: benchmark items are social and procedural products. Their defects are not random; they come from sourcing, formatting, answer-key transfer, domain drift, jurisdictional assumptions, and human annotation practices.
ImageNet reannotation and human-machine comparison
ImageNet audits show how canonical benchmarks can remain useful while still requiring relabeling and reinterpretation. Shankar et al. reannotated 40,000 ImageNet test images with trained human labelers, a detailed annotation guide, and domain expertise for difficult categories such as dog breeds; they found that many images had more than one valid label, making top-1 comparison overly pessimistic in some settings. Proceedings of Machine Learning Research
This maps closely onto MMLU’s multiple-correct-answer problem. A benchmark can appear to have a single ground truth because its file format has one label, not because the world or the question actually admits only one defensible answer.
“Fantastic Bugs” and automated invalidity detection
The “Fantastic Bugs” line of work argues that many benchmark items are invalid because of ambiguous wording, incorrect answer keys, or grading issues, and proposes model-assisted methods for finding likely defects. Its authors report high precision for some invalid-item detection workflows and emphasize that most benchmarks are rarely revised despite driving major research conclusions. Stanford AI Laboratory
MMLU-Redux is more conservative than a purely automated invalidity detector. It uses human expert re-annotation and source verification. The two approaches fit together naturally: automated systems can prioritize suspicious items, while human experts adjudicate and produce versioned corrections.
Text-to-SQL benchmark audits
Recent text-to-SQL audits also find substantial benchmark defects, including wrong ground-truth SQL, ambiguous natural-language questions, and mismatches between questions, databases, and domain knowledge. One audit reports high error rates in specific text-to-SQL benchmark variants and shows that corrected evaluation can change performance and rankings. vldb.org
The text-to-SQL case reinforces a general rule for AI Evaluation: as tasks become more semantically rich, benchmark defects become less like simple typos and more like disputes about task specification. This is why MMLU-Redux’s taxonomy is more useful than a binary “wrong/right” label.
The benchmark-maintenance problem
Static benchmarks are institutionally convenient
Static public benchmarks are easy to cite, easy to reproduce, and easy to integrate into evaluation harnesses. They are also easy to overfit, contaminate, and neglect. MMLU’s original success created a coordination point: labs, papers, leaderboards, and open-source releases could all report a familiar number. But coordination points become brittle when their underlying item pools are not maintained.
MMLU-Redux exposes the maintenance gap. The original benchmark became a de facto standard, but the ecosystem did not have an equally standard procedure for item correction, versioning, dispute resolution, or historical-score migration.
Community-driven maintenance
MMLU-Redux’s public dataset materials describe a community contribution workflow through Hugging Face discussions, including tags for additions and fixes, details for new annotations or revisions, and preservation of older versions through git commit hashes. The authors also describe update announcements and contingency availability through mirrors such as Zenodo. arXiv
Community maintenance has obvious strengths:
| Strength | Why it matters |
|---|---|
| Domain coverage | No small author team can fully cover medicine, law, chemistry, logic, and global facts. |
| Error discovery | Public users encounter edge cases during real evaluations. |
| Transparency | Open item-level disputes can be inspected and replicated. |
| Version velocity | Fixes do not need to wait for a new benchmark paper. |
But community maintenance also has risks:
| Risk | Failure mode |
|---|---|
| Inconsistent standards | Different contributors may apply different validity thresholds. |
| Reputation attacks | Public leaderboards create incentives to contest items strategically. |
| Expertise uncertainty | Not every confident correction is expert correction. |
| Version fragmentation | Multiple unofficial cleaned MMLU variants can make scores incomparable. |
A purely open edit model would be too loose for a benchmark that influences model rankings. But a purely closed model would underuse the distributed expertise needed for a benchmark with 57 domains.
Original-author ownership
The alternative is stewardship by the original benchmark authors or a small designated maintainer group. This has different strengths:
| Strength | Why it matters |
|---|---|
| Continuity | The benchmark’s intended construct is interpreted by its creators. |
| Version discipline | Official releases reduce fragmentation. |
| Accountability | Users know whose policy governs inclusion and correction. |
| Reproducibility | Historical versions can be preserved under a clear naming scheme. |
The weakness is bottlenecking. The original authors may not have the time, domain expertise, or incentive structure to maintain every subject indefinitely. For broad benchmarks, “owned by original authors forever” is not obviously scalable.
A hybrid governance model is strongest
The best governance model for MMLU-like benchmarks is community-driven evidence intake under named expert stewardship. The item pool should be publicly auditable, but official changes should be versioned, justified, and reviewed. A benchmark-maintenance process should include:
| Governance requirement | Concrete practice |
|---|---|
| Item-level provenance | Record source, annotator rationale, and correction history. |
| Typed error taxonomy | Distinguish wrong labels from ambiguity, no-answer, and multi-answer cases. |
| Expert review | Require domain-qualified adjudication for official changes. |
| Public dispute log | Keep rejected and accepted corrections visible. |
| Semantic versioning | Separate patch-level label fixes from major benchmark rewrites. |
| Historical reproducibility | Preserve old versions and publish score migration notes. |
| Contamination policy | Track which versions are public, private, or retired. |
This hybrid model is also the cleanest answer to the open question of ownership. Original authors should not be the sole permanent gatekeepers, but neither should benchmark correction become an uncurated wiki edit war. The appropriate unit of ownership is a stewardship process, not a person.
How MMLU-Redux should change evaluation practice
Report audited and unaudited scores separately
A serious MMLU report should distinguish at least three quantities:
| Score | Meaning |
|---|---|
| Original MMLU score | Historical comparability with prior work. |
| MMLU-Redux valid-item score | Performance on items judged acceptable. |
| Corrected-key score | Performance after fixing wrong labels where correction is justified. |
These are not interchangeable. Original-score reporting answers “how does this model compare to historical MMLU numbers?” Valid-item scoring answers “how does this model perform on audited items?” Corrected-key scoring answers “how much did wrong labels distort the measured result?” Combining them without labeling the protocol creates a new ambiguity.
Treat high-defect subjects differently
Virology, Logical Fallacies, College Chemistry, Professional Law, and other high-defect subjects should be flagged in evaluation reports. A subject slice with a 10–57% audited defect rate cannot be interpreted like a subject slice with negligible detected defects. arXiv
For high-defect subjects, the most defensible reporting pattern is:
| Report element | Purpose |
|---|---|
| Original subject accuracy | Maintains backward comparability. |
| Valid-item-only accuracy | Estimates construct performance on acceptable items. |
| Number of retained items | Prevents hiding small-sample effects. |
| Error-type breakdown | Shows whether the issue is labels, ambiguity, or options. |
| Rank sensitivity | Shows whether conclusions change under cleaning. |
Do not use MMLU-Redux as a license to cherry-pick
A cleaned benchmark can be abused. If evaluators remove only items their model gets wrong, or selectively apply corrections, they turn benchmark maintenance into post-hoc score optimization. That is why item-level correction policies must be fixed before evaluation, versioned, and applied uniformly across models.
The ethical rule is simple: item cleaning should improve measurement validity, not model marketing.
Combine with contamination-aware evaluation
MMLU-Redux does not solve contamination. MMLU-CF does not fully solve item validity. MMLU-Pro does not guarantee clean labels just because it is harder. Strong evaluation suites should combine audited public benchmarks, contamination-resistant private benchmarks, dynamic evaluation, and domain-specific expert review. arXiv
A practical evaluation stack might look like:
| Layer | Example | Role |
|---|---|---|
| Historical comparability | Original MMLU | Continuity with prior reports. |
| Item-quality audit | MMLU-Redux-filtered MMLU | Validity check on public benchmark items. |
| Harder reasoning benchmark | MMLU-Pro | Better discrimination among stronger models. |
| Contamination-resistant benchmark | MMLU-CF | Reduced exposure and overfitting risk. |
| Domain expert eval | Custom medicine, law, code, or safety tasks | Task-specific validity and deployment relevance. |
What MMLU-Redux does not settle
MMLU-Redux answers an important empirical question: many MMLU items have identifiable correctness or ambiguity problems. It does not answer every evaluation question raised by modern LLMs.
First, it does not fully audit the entire original MMLU test set. The public audit covers 5,700 sampled items, and the paper explicitly identifies the remaining 8,342 items as future annotation work. arXiv
Second, it does not produce a universally official replacement benchmark. It gives the community data needed to construct cleaned variants, but official adoption depends on governance choices.
Third, it does not resolve the construct-validity question of whether multiple-choice exams are the right way to evaluate advanced AI systems. Even a perfectly labeled MMLU item still measures only a narrow interaction: choose one of four answer options under a particular prompting and scoring protocol.
Fourth, it does not solve temporal drift. Subjects such as global facts, law, medicine, and machine learning can change over time. A valid item in 2020 may become outdated in 2026. Benchmark maintenance is therefore not a one-time cleanup but a recurring obligation.
Source landmarks
| Source | Why it matters |
|---|---|
| MMLU original paper | Establishes the 57-subject benchmark and its intended role as a broad test of knowledge and problem solving. arXiv |
| MMLU-Redux paper | Primary source for the 5,700-item audit, taxonomy, subject-level error findings, ranking-sensitivity experiments, and 6.49% full-test defect estimate. arXiv |
| MMLU-Redux dataset card | Documents the public dataset structure, license, subject subsets, and audit columns. Hugging Face |
| MMLU-Pro paper and repository | Primary source for the “harder, more reasoning-focused” remediation strategy: more answer choices, curated questions, lower accuracies, and reduced prompt sensitivity. arXiv |
| MMLU-CF paper and project materials | Primary source for contamination-resistant evaluation, closed test policy, decontamination rules, and reported performance/rank changes. arXiv |
| Pervasive Label Errors | Establishes that label errors in canonical test sets can destabilize model selection and are not unique to LLM benchmarks. arXiv |
| Annotation Artifacts in NLI | Shows how dataset construction artifacts can create shortcut behavior and overstate benchmark progress. ACL Anthology |
| Fantastic Bugs | Connects MMLU-Redux to broader work on invalid benchmark items and model-assisted benchmark auditing. Stanford AI Laboratory |
| Text-to-SQL benchmark audits | Shows similar item-validity failures in semantically rich structured-output benchmarks. vldb.org |
Companion entries
Core theory: Benchmark Validity, Measurement Error in Benchmarks, Dataset Auditing, Annotation Artifacts, Item Response Theory for LLM Evaluation
MMLU family: MMLU, MMLU-Pro, MMLU-CF, Global MMLU, HELM
Evaluation practice: Leaderboard Hygiene, Benchmark Maintenance, Contamination-Resistant Evaluation, Model Comparison Under Measurement Noise, Evaluation Harnesses
Governance: Community Governance of AI Benchmarks, Benchmark Versioning, Open Benchmark Stewardship, Closed Test Sets
Counterarguments and limitations: Benchmark Saturation, Multiple-Choice Evaluation Limits, Benchmark Contamination, Goodhart’s Law in AI Evaluation