MMLU-Pro is best understood as a methodological repair of MMLU, not merely a harder leaderboard: Wang et al. kept MMLU’s broad academic and professional ambition while replacing weak measurement elements—four-option recognition, easy or exhausted items, thin distractors, and noisy labels—with a filtered, refreshed, ten-option, reasoning-weighted benchmark. Its initial value was discriminative headroom: frontier models that bunched together on MMLU spread out on MMLU-Pro, and chain-of-thought prompting mattered much more. By 2026, however, public leaderboards already show frontier systems pushing into roughly the 90% range, making MMLU-Pro’s long-term value depend on version discipline, contamination controls, and complementary variants such as MMLU-Pro+, MMLU-CF, and MMLU-Redux.
Coverage note: verified through May 12, 2026.
Why MMLU-Pro exists
The original MMLU benchmark, introduced by Hendrycks et al., was designed to measure broad multitask knowledge and problem-solving across 57 subjects, including STEM, humanities, social sciences, and professional domains. Its test split contains 14,042 questions, each with four answer choices, making random guessing worth 25% before any language understanding enters the picture. At release, MMLU still had substantial headroom: the benchmark card reports GPT-3 at 43.9%, UnifiedQA at 48.9%, and a random baseline at 25%. arXiv
MMLU’s success created the conditions for its decline as a frontier-model discriminator. Once a benchmark becomes canonical, it becomes part of model-development culture: papers report it, data pipelines screen against it imperfectly, prompt templates optimize for it, and leaderboard differences shrink. Wang et al.’s MMLU-Pro was created to recover some of the measurement pressure MMLU originally had: the benchmark adds reasoning-heavy questions, expands choices from four to up to ten, removes trivial or noisy items, and verifies questions more aggressively. The authors report that MMLU-Pro caused a 16–33 percentage-point drop relative to MMLU across evaluated models, while reducing prompt sensitivity from roughly 4–5% on MMLU to around 2% on MMLU-Pro. NeurIPS Papers
One subtle but important correction: MMLU-Pro does not preserve MMLU’s original 57-subject reporting scheme one-to-one. It preserves MMLU’s broad academic and professional coverage goal, but merges the subject space into 14 broader domains and adds a large number of refreshed items from sources beyond MMLU. The Hugging Face benchmark card lists 12,032 total questions, with 6,810 originating from MMLU and 5,222 newly added; the domains include biology, business, chemistry, computer science, economics, engineering, health, history, law, math, philosophy, physics, psychology, and “other.” Hugging Face
Design at a glance
| Benchmark | Main purpose | Subject structure | Answer format | Approximate scale | Contamination / quality posture |
|---|---|---|---|---|---|
| MMLU | Broad multitask academic and professional knowledge | 57 subjects | 4-choice multiple choice | 14,042 test questions | Historically important, but now heavily exposed and label-quality issues have been documented |
| MMLU-Pro | Less-saturated successor with more reasoning pressure | 14 broad domains derived from and extending MMLU coverage | Mostly 10-choice multiple choice; average 9.47 options after review | 12,032 questions | Filters easy items, adds refreshed items, augments distractors, and performs expert review |
| MMLU-Pro+ | Shortcut-learning and multi-answer reasoning stress test | Built on MMLU-Pro | Multiple correct answers plus crafted distractor categories | Derived from MMLU-Pro | Tests anchoring, shortcut selection, and ability to identify correct answer pairs |
| MMLU-CF | Contamination-free MMLU-style evaluation | Broad MMLU-like disciplinary coverage | Multiple choice | 10k validation + 10k closed test | Uses decontamination rules, public validation, and closed-source test split |
| MMLU-Redux | Label-quality repair for MMLU | Original MMLU subject space | 4-choice multiple choice with reannotations | 5,700 manually reannotated questions in Redux 2.0 | Audits and corrects wrong labels, ambiguous questions, and invalid options |
The core measurement change is simple: MMLU-Pro makes the same general family of task harder to solve by shallow recognition. A four-option question often lets a model succeed by eliminating two implausible distractors; a ten-option question with harder distractors forces finer discrimination. The MMLU-Pro paper reports that 83% of retained questions have ten options, and the remaining 17% have fewer because expert review removed problematic choices; the average retained option count is 9.47. arXiv
That change also alters the random baseline. MMLU’s nominal random-choice baseline is 25%; MMLU-Pro’s practical random baseline is near 10%, though not exactly 10% because not every question retains ten choices after review. This matters because multiple-choice benchmarks partly measure answer-space search: a model that is only weakly calibrated can look better when the answer space is small, especially if distractors are stylistically or semantically weak. arXiv
Construction methodology
1. Subject consolidation and item filtering
MMLU-Pro begins with MMLU’s subject matter but changes the reporting and item-selection geometry. Wang et al. merge the original 57 MMLU subjects into 14 broader categories, then run an initial filtering pass over the original MMLU items using eight models. Questions answered correctly by more than four of those models were treated as too easy and removed. Across the source set used in this filtering pass, 5,886 of 13,937 questions were filtered out, or 42.23%. arXiv
This is the first major methodological contribution: MMLU-Pro explicitly removes low-information items. In benchmark terms, an item that nearly all current models answer correctly contributes little to ranking frontier systems, even if it remains useful for testing weaker models. Filtering such items restores some discriminative capacity without needing to invent an entirely new task family.
2. Item refresh from external sources
MMLU-Pro is not merely “MMLU with more answer choices.” After filtering, the authors add new questions from the STEM Website, TheoremQA, and SciBench, while retaining a large subset of filtered-and-verified MMLU questions. The reported final source composition is 6,810 MMLU-derived questions, 4,083 STEM Website questions, 598 TheoremQA questions, and 541 SciBench questions, totaling 12,032. arXiv
The refresh matters for two reasons. First, it replaces stale or overexposed items with questions less likely to have become implicit training targets. Second, it shifts the benchmark toward reasoning-heavy material, especially in math, science, and theorem-style questions. This does not make MMLU-Pro contamination-proof—its dataset is public—but it does make the benchmark less dependent on the exact aging MMLU item pool.
3. Harder distractors and ten-option expansion
The most visible change is option augmentation. MMLU-Pro uses GPT-4-Turbo to expand four-option questions toward ten options by generating additional plausible distractors; these are then subjected to review. For questions imported from sources without the same multiple-choice structure, GPT-4-Turbo was also used to extract or construct candidate multiple-choice forms. arXiv
The design goal is not merely a lower chance baseline. Ten options with weak distractors would be cosmetic. The intended measurement pressure comes from “hard distractors”: options close enough to the correct answer that a model must reason through the underlying domain content rather than rely on answer-style artifacts. This is particularly relevant for modern instruction-tuned models, which often learn the surface texture of benchmark answers.
4. Expert verification and false-negative control
Wang et al. use a two-stage expert-review process. In the first phase, experts verify question correctness, remove questions unsuitable for multiple-choice evaluation, remove non-textual or under-specified items, and check whether the answer key is accurate. In the second phase, Gemini-1.5-Pro is used to flag possible false negatives—cases where an allegedly incorrect option may also be valid—and human experts then review those flagged items. arXiv
The error counts reported by the authors illustrate why this matters. In the reviewed source pools, they found hundreds of incorrect answers and many more false-negative options, especially in MMLU-derived and newly generated-option settings. The table in the paper reports, for example, 350 incorrect-answer issues and 1,953 false-negative-option issues in the MMLU-derived pool, as well as 483 incorrect-answer issues and 293 false-negative-option issues in the STEM Website-derived pool. arXiv
This is a crucial point for practitioners: harder benchmarks often become worse benchmarks if they merely add adversarial or ambiguous questions. MMLU-Pro’s construction methodology tries to separate “hard because it requires reasoning” from “hard because the label or wording is broken.” The separation is imperfect, but the explicit expert-verification layer is part of why MMLU-Pro is more than a difficulty hack.
What MMLU-Pro measures better than MMLU
MMLU-Pro rebuilds MMLU’s discriminative power along three axes: answer-space pressure, reasoning depth, and prompt robustness.
First, expanding from four options to mostly ten options reduces the value of shallow elimination. In a four-choice setting, a model may perform well by recognizing that two choices are implausible and then picking between the remaining two using memorized associations. In a ten-choice setting with harder distractors, the model has to maintain more distinctions at once. That does not transform multiple choice into open-ended reasoning, but it raises the floor for superficial success. arXiv
Second, MMLU-Pro makes chain-of-thought prompting more consequential. Wang et al. report that GPT-4o improved from 53.5% direct-answer accuracy to 72.6% with chain-of-thought on MMLU-Pro, a 19.1-point gain; on MMLU, the same comparison was 87.2% to 88.7%, only a 1.5-point gain. GPT-4-Turbo showed a similar pattern: roughly no gain on MMLU, but a 15.3-point gain on MMLU-Pro. arXiv
Third, MMLU-Pro appears less prompt-fragile than MMLU under the paper’s evaluation. Wang et al. evaluated 24 prompt variants and found that MMLU scores could vary by roughly 4–5% and peak near 10.98% prompt-induced variation, while MMLU-Pro generally stayed around 2% variation with a reported maximum of 3.74%. arXiv
The important methodological claim is therefore not “MMLU-Pro is harder.” Hardness alone is cheap. The better claim is: MMLU-Pro restores ranking resolution among models that MMLU had begun to compress, and it does so while making reasoning-oriented prompting more useful and prompt-template artifacts less dominant.
Empirical trajectory across frontier models
The 2024 baseline: restored separation
In the MMLU-Pro paper, GPT-4o scored 72.6%, Gemini-1.5-Pro scored 69.0%, Claude-3-Opus scored 68.5%, GPT-4-Turbo scored 63.7%, Gemini-1.5-Flash scored 59.1%, Claude-3-Sonnet scored 56.8%, and Llama-3-70B-Instruct scored 56.2%. The same paper emphasizes that MMLU-Pro widened model gaps that were narrow on MMLU; for example, GPT-4o and GPT-4-Turbo were separated by roughly 9 points on MMLU-Pro but about 1 point on MMLU. arXiv
The GitHub leaderboard associated with the benchmark later listed Claude-3.5-Sonnet at 76.12%, GPT-4o at 72.55%, Gemini-1.5-Pro at 69.03%, Claude-3-Opus at 68.45%, and GPT-4-Turbo at 63.71%. These numbers made the initial practical story clear: MMLU-Pro did not merely lower every model’s score uniformly; it separated models that had looked nearly tied on MMLU. GitHub
The 2026 picture: approaching saturation again
By May 2026, public MMLU-Pro leaderboards show a different picture. One leaderboard drawing from Artificial Analysis data lists top MMLU-Pro scores around 89–90%, with Gemini 3 Pro Preview at 89.8%, Claude Opus 4.5 Thinking at 89.5%, and other frontier systems close behind; the same page reports 218 tracked models, an average score of 74.9%, and a standard deviation of 11.5. Price Per Token
A separate Vals.ai leaderboard, updated May 4, 2026, reports an even higher top score: Gemini 3.1 Pro Preview at 90.99%, followed by Gemini 3 Pro at 90.10% and Claude Opus 4.7 at 89.87%. Vals explicitly characterizes MMLU-Pro as “largely saturated” in its benchmark description, while also stating that reported results use 5-shot chain-of-thought, matching the original MMLU-Pro evaluation style. vals.ai
These leaderboards should be treated as trajectory evidence, not as a single controlled experiment. Model snapshots change, providers expose different inference modes, “thinking” models may use different reasoning budgets, and public leaderboards often mix internal runs, submitted results, and third-party harnesses. The right conclusion is not that every frontier model is now reliably at 90% under one fixed protocol; it is that MMLU-Pro’s original headroom has already narrowed substantially at the top.
The methodology contribution: rebuilding discriminative power
MMLU-Pro’s main contribution is methodological rather than taxonomic. It shows how to extend a saturated benchmark while preserving continuity with the older one:
| Saturation problem in MMLU | MMLU-Pro intervention | Measurement effect |
|---|---|---|
| Many items too easy for frontier models | Filter items solved by most of eight reference models | Removes low-information questions |
| Four answer choices enable shallow elimination | Expand toward ten options | Reduces random baseline and cheap elimination |
| Weak distractors reveal the answer | Generate harder distractors, then review | Forces finer semantic discrimination |
| Aging public item pool | Add refreshed questions from external sources | Reduces dependence on stale MMLU items |
| Label errors and false negatives | Expert review plus model-assisted false-negative detection | Improves validity of hard questions |
| Prompt-template sensitivity | Evaluate multiple prompt variants | Tests whether score differences survive prompt changes |
This is a useful pattern for benchmark maintenance. Static benchmarks do not fail only because models improve; they also fail because their item distribution becomes culturally and technically overexposed. MMLU-Pro responds by refreshing the item pool, increasing item difficulty, and auditing labels. That combination is more durable than simply adding “hard” questions.
The paper’s error analysis also matters. Wang et al. report that among GPT-4o’s MMLU-Pro errors, 39% involved reasoning errors, 35% involved domain-expertise gaps, and 12% involved computational errors. That distribution supports the claim that MMLU-Pro is measuring more than memorized factual recall, though it remains a multiple-choice benchmark and therefore cannot fully test open-ended synthesis, research planning, or creative problem-solving. arXiv
Variants and related refreshes
MMLU-Pro+
MMLU-Pro+ is a direct descendant of MMLU-Pro aimed at shortcut learning and higher-order reasoning. Rather than assuming exactly one correct answer, MMLU-Pro+ introduces questions with multiple correct answers and crafted distractor categories, then evaluates whether a model can distinguish fully correct, partially misleading, and completely false answer pairs. arXiv
The MMLU-Pro+ paper describes three answer-pair categories: True Positive Pairs, Partial False Positive Pairs, and Complete False Pairs. It reports modifying 3,718 questions to include true positive pairs, alongside 2,029 complete false-positive and 2,124 partial false-positive constructions. The stated goal is to reduce shortcut learning by forcing the model to recognize when more than one option is valid and when a plausible option contains a subtle error. ar5iv
MMLU-Pro+ also introduces metrics beyond ordinary accuracy. Shortcut Selection Ratio is intended to capture anchoring on the original or most obvious correct answer, while Correct Pair Identification Ratio measures the ability to identify valid pairs. In its evaluation, the paper reports average scores such as 68.3 for O1-preview, 67.9 for Sonnet-3.5, 62.6 for Llama-405B, and 60.6 for GPT-4o, with model-specific differences in shortcut behavior. ar5iv
The practical interpretation: MMLU-Pro+ is not simply “MMLU-Pro but harder.” It asks a different behavioral question: can the model avoid single-answer anchoring when a domain problem admits multiple valid answers or subtly invalid alternatives? That makes it especially relevant for Shortcut Learning in LLMs and Multiple-Answer Evaluation, but less directly comparable to standard MMLU or MMLU-Pro accuracy.
MMLU-CF
MMLU-CF addresses a different failure mode: contamination. The ACL 2025 paper presents it as a contamination-free and challenging multiple-choice benchmark, designed to prevent both unintentional and malicious benchmark leakage. It sources questions from more than 200 billion webpages, applies three decontamination rules, provides a public validation set, and keeps the test set closed-source. ACL Anthology
The Hugging Face card describes MMLU-CF as having 10,000 validation and 10,000 test questions, with a pipeline including MCQ collection, cleaning, difficulty sampling, LLM checking by GPT-4o, Gemini, and Claude, and contamination-free processing. It explicitly frames MMLU-CF as a response to the fact that MMLU’s breadth and reasoning coverage do not, by themselves, prevent contamination. Hugging Face
This makes MMLU-CF important but operationally different. A closed test set improves contamination resistance, but reduces fully open reproducibility. It is most appropriate when the evaluation claim is about leakage-resistant generalization rather than easy-to-reproduce open benchmarking.
MMLU-Redux
MMLU-Redux is not a successor in the same sense as MMLU-Pro. It is a label-quality repair effort for MMLU. The NAACL 2025 paper “Are We Done with MMLU?” reports numerous ground-truth errors, including the striking finding that 57% of analyzed virology questions contained errors, and introduces MMLU-Redux as a manually reannotated subset across MMLU subjects. ACL Anthology
The MMLU-Redux 2.0 dataset card lists 5,700 rows and preserves fields such as the original answer, corrected answer, error type, and annotator rationale. Its taxonomy includes bad question clarity, bad options clarity, no correct answer, multiple correct answers, and wrong ground truth. Hugging Face
MMLU-Redux is valuable because benchmark saturation and benchmark invalidity are different problems. MMLU-Pro addresses saturation by increasing difficulty and refreshing the item pool; MMLU-Redux addresses reliability by auditing whether original MMLU labels and options were valid. A serious evaluation program should care about both.
Which MMLU variant should practitioners report?
There is no single best MMLU-family number anymore. The right benchmark depends on the claim being made.
| Evaluation claim | Best primary report | Useful secondary report | Main caveat |
|---|---|---|---|
| “This model is comparable to older papers.” | Original MMLU | MMLU-Redux where available | Historical comparability comes at the cost of saturation and possible label errors |
| “This model improves broad academic/professional reasoning.” | MMLU-Pro | Original MMLU for continuity | Must specify prompt, shots, chain-of-thought policy, split, and scoring harness |
| “This model is robust against benchmark leakage.” | MMLU-CF | MMLU-Pro | Closed test improves contamination resistance but complicates independent reproduction |
| “This model avoids shortcut learning and single-answer anchoring.” | MMLU-Pro+ | MMLU-Pro | Multi-answer setup is not directly comparable to standard accuracy |
| “This model’s MMLU score is label-reliable.” | MMLU-Redux | Original MMLU | Redux measures corrected subsets, not the full historical benchmark |
For frontier-model reporting, the most defensible default is a small panel rather than one score: original MMLU for historical continuity, MMLU-Pro for current broad reasoning pressure, and either MMLU-CF or another hidden/contamination-controlled benchmark for leakage-sensitive claims. If the paper is specifically about shortcut learning, add MMLU-Pro+; if the paper is about benchmark validity or label noise, add MMLU-Redux.
The reporting standard should include: benchmark name and version, number of shots, whether chain-of-thought was used or hidden, prompt template, answer-extraction method, model snapshot, decoding parameters, option ordering, evaluation harness, date of evaluation, and whether any decontamination was performed. Without those details, MMLU-family numbers are often not comparable across papers.
The cross-paper comparability problem
MMLU-Pro improves the benchmark, but it does not eliminate evaluation-harness variance. The original MMLU-Pro paper uses a 5-shot chain-of-thought prompt and an answer-extraction procedure, with fallback behavior when an answer cannot be extracted. The Hugging Face card notes that leaderboard results are normally run under 5-shot settings, but that some Gemini submissions were run 0-shot. arXiv
That detail alone can invalidate naive comparisons. If one paper reports 5-shot chain-of-thought MMLU-Pro and another reports 0-shot direct-answer MMLU-Pro, the numbers are measuring different protocols. The gap is not trivial: Wang et al. report large CoT gains on MMLU-Pro for several frontier models. arXiv
The problem is amplified by reasoning models. A “thinking” mode with a large hidden reasoning budget may be closer to test-time search than ordinary next-token answering. If a leaderboard mixes ordinary chat models, reasoning models, and models with provider-specific inference scaffolding, the benchmark still reports a single score, but the intervention being measured is no longer just pretraining or post-training quality.
The practical rule: never cite “MMLU-Pro = X%” without the protocol. A rigorous paper should say something like: “MMLU-Pro, 5-shot CoT, official prompt, official evaluation script, model snapshot dated YYYY-MM-DD, temperature 0, evaluated on full test split, no tool use, no retrieval.” Without that, cross-paper comparability is weak.
Will MMLU-Pro follow MMLU’s saturation arc?
The evidence points in both directions, but the stronger bet is that public static MMLU-Pro will saturate for frontier systems.
The pro-saturation evidence is already visible. In 2024, GPT-4o’s MMLU-Pro score around 72.6% left meaningful headroom. By May 2026, public leaderboards report top scores near 90–91%, and at least one leaderboard describes the benchmark as largely saturated. arXiv+2Price Per Token+2
The anti-saturation evidence is that MMLU-Pro still separates many non-frontier systems. The Artificial Analysis-derived leaderboard reports 218 tracked models with an average of 74.9 and standard deviation of 11.5, which means the benchmark still has broad practical ranking value outside the very top cluster. Price Per Token
The deeper issue is that saturation is not binary. A benchmark can be saturated for frontier closed models, useful for open-model regression testing, informative for domain-level error analysis, and inadequate for contamination-resistant claims—all at the same time. That is likely where MMLU-Pro is heading.
The long-term role of MMLU-Pro should therefore be as part of an evaluation stack, not as the successor-to-end-all-successors. Its strongest use is to maintain continuity with MMLU while restoring some reasoning pressure. Its weakest use is as a sole headline number for frontier intelligence.
Takeaways
MMLU-Pro is an important benchmark because it demonstrates a disciplined way to repair a saturated benchmark while preserving enough continuity for historical comparison. It filters easy items, refreshes the item pool, increases the answer-space difficulty, adds harder distractors, and performs expert verification. That combination makes it more valuable than a benchmark that merely adds adversarial questions.
Its empirical history also contains a warning. MMLU-Pro restored separation in 2024, but by 2026 top public scores were already approaching the low 90s. The benchmark may remain useful for broad model comparison, domain analysis, and regression testing, but it is unlikely to remain a definitive frontier-capability measure unless paired with contamination controls, hidden test sets, dynamic item generation, multi-answer reasoning tests, and open-ended evaluations.
The best practitioner stance is version pluralism with explicit protocols: report MMLU for historical continuity, MMLU-Pro for less-saturated broad reasoning, MMLU-CF for contamination-sensitive claims, MMLU-Pro+ for shortcut and multi-answer behavior, and MMLU-Redux when label reliability is central.
Companion entries
Core theory:
Multiple-Choice Evaluation Geometry
Chain-of-Thought Evaluation
Data Contamination in LLM Benchmarks
Shortcut Learning in LLMs
Evaluation Validity vs Evaluation Difficulty
MMLU family:
MMLU-ProX
Adjacent benchmarks:
BIG-Bench Hard
TheoremQA
SciBench
OpenBookQA
Practice:
LLM Evaluation Harnesses
Benchmark Reporting Checklists
Model Card Evaluation Tables
Contamination-Aware Evaluation
Prompt Sensitivity in Benchmarks
Reasoning Budget Normalization
Counterarguments:
Are Static Benchmarks Obsolete?
Open-Ended vs Multiple-Choice Evaluation
Leaderboard Overfitting
Benchmark Leakage
Hidden Test Sets and Reproducibility