MMMU is a college-level multimodal benchmark designed to test whether vision-language models can combine visual perception, domain knowledge, and reasoning across academic disciplines rather than merely caption images or answer everyday visual questions. Its central claim is that expert-level multimodal understanding requires models to solve image-grounded problems in science, medicine, engineering, humanities, business, and art/design—making MMMU a multimodal analogue of MMLU rather than another general VQA benchmark.
Coverage note: verified through May 19, 2026.
Snapshot
| Property | MMMU design |
|---|---|
| Full name | Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark |
| Primary paper | Yue et al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” CVPR 2024 (Yue et al. 2024 — CVPR) |
| Task type | College-level image-grounded questions requiring perception, knowledge, and reasoning |
| Dataset size | 11,550 questions |
| Splits | 150 development, 900 validation, 10,500 test |
| Subject coverage | 6 broad disciplines, 30 subjects, 183 subfields |
| Image coverage | 30 heterogeneous image types in the paper’s accounting; the GitHub README currently says 32, a minor documentation discrepancy |
| Format | Mostly multiple-choice, with a smaller open-answer component |
| Original frontier baseline | GPT-4V around 56% accuracy; Gemini Ultra and Claude 3 Opus reported around 59% in later comparisons |
| Human reference | Human expert bands reported around 76.2%, 82.6%, and 88.6% for worst/medium/best settings |
| Success criterion | Not “can the model see?” alone, but “can it use images as evidence inside disciplinary reasoning?” |
The paper reports 11,550 questions spanning six disciplines, 30 subjects, 183 subfields, and 30 image types, with GPT-4V at roughly 56% versus much higher expert-human performance (Yue et al. 2024 — CVPR). openaccess.thecvf.com+2openaccess.thecvf.com+2
Why MMMU exists: MMLU, but images matter
MMLU made “broad college-level knowledge” a standard axis for evaluating language models. MMMU extends that idea into Multimodal Reasoning: a model is not merely asked to classify an image or answer a trivia question with a picture attached; it is asked to solve academic problems where diagrams, plots, medical scans, maps, art works, design layouts, chemical structures, tables, and other visual artifacts are part of the evidence.
That distinction matters. Many earlier multimodal benchmarks could be solved by matching surface image-text associations, reading visible text, or relying on common-sense priors. MMMU’s authors explicitly frame the benchmark as a response to the gap between everyday multimodal VQA and expert-level tasks that require “perception, knowledge, and reasoning” together. The paper contrasts MMMU with benchmarks centered on daily images or common-sense reasoning and emphasizes college-level subject coverage as the differentiator (Yue et al. 2024 — CVPR). openaccess.thecvf.com
MMMU’s “multimodal MMLU” role is therefore conceptual, not just cosmetic. MMLU asks whether a language model has broad disciplinary competence in text. MMMU asks whether a multimodal model can use disciplinary visual artifacts as first-class inputs. A question about an electrical circuit, a pathology image, a musical score, a geological map, or a statistical chart is not reducible to generic object recognition; the model must interpret the visual representation in the grammar of a field.
Benchmark design
The core unit: an image-grounded academic problem
A MMMU item usually consists of a question, one or more images, and answer options or a short open-ended answer target. The image is not decorative. In the authors’ dataset statistics, images appear in 97.52% of questions, images appear in options for 3.37% of examples, and 7.39% of examples contain multiple images (Yue et al. 2024 — CVPR). openaccess.thecvf.com
The benchmark is designed around four challenges: broad subject coverage, heterogeneous image types, interleaved text-image inputs, and expert-level perception plus reasoning. The examples shown in the paper include tasks such as interpreting scientific diagrams, medical figures, charts, maps, and disciplinary visual conventions rather than everyday photographs alone (Yue et al. 2024 — CVPR). openaccess.thecvf.com
This makes MMMU a test of compound competence:
| Competence | What it means in MMMU |
|---|---|
| Visual perception | Identify relevant visual structures: labels, axes, symbols, colors, geometry, spatial relations, anatomical regions, visual encodings |
| Domain knowledge | Know the disciplinary meaning of the visual structure: what a circuit component, chemical diagram, medical image, musical notation, or art-historical feature implies |
| Reasoning | Combine image evidence and text constraints to infer an answer |
| Format robustness | Produce a valid multiple-choice answer or concise open answer under a specified prompt |
The benchmark’s central methodological bet is that a model cannot score well merely by being a strong language model with a weak image front end. MMMU’s reported text-only baselines support that view: GPT-4 text scored about 34.9% on validation and 33.8% on test, far below GPT-4V’s roughly 56% result; adding OCR or captions to text-only pipelines offered only limited improvement in the authors’ experiments (Yue et al. 2024 — CVPR). openaccess.thecvf.com
Dataset surface
| Statistic | Reported value |
|---|---|
| Total questions | 11,550 |
| Development set | 150 |
| Validation set | 900 |
| Test set | 10,500 |
| Disciplines | 6 |
| Subjects | 30 |
| Subfields | 183 |
| Image types | 30 in the paper/site accounting |
| Multiple-choice questions | 94.03% |
| Open-ended questions | 5.97% |
| Images in question | 97.52% |
| Images in options | 3.37% |
| Multiple images per item | 7.39% |
| Difficulty distribution | 28% easy, 45% medium, 27% hard |
These statistics come from the MMMU paper’s dataset table and data description. The GitHub repository now also notes that the test-set answers were released on February 12, 2026, so current local evaluation differs from the original hidden-test leaderboard setting (MMMU GitHub). openaccess.thecvf.com
Discipline coverage
MMMU’s six broad disciplines are intended to make the benchmark difficult to overfit to one visual domain. The paper organizes questions across art and design, business, science, health and medicine, humanities and social science, and technology and engineering. Within those areas, it covers 30 subjects and 183 subfields, including domains where visual representations are central rather than incidental (Yue et al. 2024 — CVPR). openaccess.thecvf.com
| Discipline family | Why visual input matters |
|---|---|
| Art and design | Images, layouts, style, composition, perspective, and visual culture are primary evidence |
| Business | Charts, tables, decision diagrams, and quantitative displays are common |
| Science | Diagrams, plots, chemical structures, microscope images, physical systems, and lab figures are integral |
| Health and medicine | Medical images, anatomical diagrams, pathology figures, and clinical visual cues matter |
| Humanities and social science | Maps, historical artifacts, artworks, charts, and diagrams can serve as evidence |
| Technology and engineering | Circuits, mechanical diagrams, architecture, code-related visuals, CAD-like figures, and system schematics are frequent |
The important point is not simply breadth. The benchmark chooses domains where images carry formal information. A geometry diagram, a chemical structure, a medical scan, and an art-history slide all require different visual literacies. MMMU tries to make that heterogeneity part of the score rather than a nuisance variable.
Construction methodology
Authorship and collection
MMMU is not a generic web-scraped VQA set. The authors report that more than 50 college students, including coauthors, curated the dataset. They selected majors in which visual inputs are common and excluded fields such as law and linguistics where the benchmark authors judged visual input to be less central. Questions were drawn from college exams, quizzes, textbooks, online resources, and newly created items where needed; annotators were also instructed around copyright and licensing constraints (Yue et al. 2024 — CVPR). openaccess.thecvf.com
“Expert-authored” should be read carefully. MMMU is expert-oriented and specialist-curated, but the paper’s construction process is primarily based on trained student/coauthor authorship rather than a fully professional panel of credentialed domain experts for every item. The benchmark’s human evaluation later uses senior undergraduate and graduate students as “human expert” references, which is reasonable for college-level questions but not equivalent to certified professional expertise in every field (Yue et al. 2024 — CVPR). openaccess.thecvf.com
Image-grounding
The construction process emphasized image-grounded questions: the image should be necessary or strongly relevant to answering the item. This is critical because many multimodal benchmarks leak into text-only reasoning if the question can be answered from the prompt alone. MMMU’s authors explicitly mention concerns about data contamination and instructed annotators to select questions whose answers were not immediately available on the internet, while also performing duplicate checks by lexical overlap and source URL (Yue et al. 2024 — CVPR). openaccess.thecvf.com
That said, image-grounding is not binary. Some questions may still contain textual or domain cues sufficient for partial guessing, especially in multiple-choice format. This is one reason MMMU-Pro later filters out items answerable by text-only models and adds harder answer formats.
Quality control
The dataset went through filtering for duplicates, typos, formatting errors, difficulty, and overly easy questions. The paper says the initial pool was around 13,000 questions and that very easy questions were removed, yielding the final 11,550-item benchmark. Difficulty labels are distributed across easy, medium, and hard categories, with medium being the largest share (Yue et al. 2024 — CVPR). openaccess.thecvf.com
The paper’s limitation section acknowledges that manual curation can introduce biases and that college-level subjects are not sufficient for a complete measure of “Expert AGI.” This is an important caveat: MMMU is a serious benchmark for expert-like multimodal reasoning, not a universal definition of expertise (Yue et al. 2024 — CVPR). openaccess.thecvf.com
Evaluation protocol
MMMU uses accuracy as the main metric. For multiple-choice questions, models are prompted to select an option; for open-ended questions, the authors use rule-based answer extraction and treat invalid open answers as incorrect. In the original evaluation, invalid multiple-choice outputs could fall back to random selection, a detail that matters when comparing models with different instruction-following reliability (Yue et al. 2024 — CVPR). openaccess.thecvf.com
The benchmark uses micro-averaged accuracy across examples. That makes the headline score easy to compare, but it also hides discipline-level and image-type-level differences. A model could be strong on charts and weak on medical images, or strong on visible text and weak on geometry, while still reporting a single aggregate score.
Evaluation details also matter because multimodal models are highly prompt-sensitive. Chain-of-thought prompting, answer-extraction rules, image resolution, option ordering, model temperature, tool access, and reasoning budget can change results. This is not unique to MMMU, but MMMU’s mixture of images, disciplines, and formats makes protocol drift especially consequential for leaderboard interpretation.
Empirical performance trajectory
Launch-era results: GPT-4V near 56%, humans far above
The original MMMU paper placed GPT-4V at about 56.8% validation accuracy and 55.7% test accuracy. The same results table reported human expert bands around 76.2%, 82.6%, and 88.6% for worst, medium, and best expert settings. Claude 3 Opus and Gemini Ultra were later listed around 59.4% in the authors’ leaderboard-style comparisons, although those values should be interpreted in the context of the specific evaluation protocols and result sources (Yue et al. 2024 — CVPR; MMMU project site). openaccess.thecvf.com
The launch-era pattern was stark: text-only and early open-source multimodal systems lagged well behind GPT-4V, while GPT-4V still lagged far behind human experts. The paper reports that GPT-4 text alone scored around the mid-30s, and that OCR/caption augmentation did not close the gap, suggesting that MMMU was not reducible to text extraction plus language-model reasoning (Yue et al. 2024 — CVPR). openaccess.thecvf.com
Error profile: perception, knowledge, and reasoning all fail
The authors manually analyzed 150 GPT-4V errors. Their categories included perceptual errors, lack of knowledge, reasoning errors, textual-understanding errors, refusals, annotation errors, and answer-extraction issues. The largest buckets were perceptual failure, missing domain knowledge, and reasoning failure, which is exactly the intended triad of the benchmark (Yue et al. 2024 — CVPR). openaccess.thecvf.com
The difficulty split also revealed the benchmark’s shape. GPT-4V scored much higher on easy items than on hard items, with the paper reporting approximately 76.1% on easy, 55.6% on medium, and 31.2% on hard. The authors also note poor performance on some image types such as geometry, music, and chemistry representations, showing that aggregate scores can conceal sharply different visual-literacy failures (Yue et al. 2024 — CVPR). openaccess.thecvf.com
2024–2025: frontier systems cross the high-60s and low-70s
By the GPT-4o, Claude 3.5 Sonnet, Gemini 1.5/2.x, and Qwen2-VL/Qwen2.5-VL era, MMMU scores moved into the high-60s and low-70s on commonly cited validation-style protocols. The MMMU-Pro paper reports original-MMMU validation scores such as GPT-4o at 69.1, Claude 3.5 Sonnet at 68.3, Gemini 1.5 Pro at 65.8, and Qwen2-VL-72B at 64.5 in its comparison table (Yue et al. 2025 — MMMU-Pro). arXiv
Third-party registries also show 70%+ MMMU results for later multimodal systems, including Qwen2.5-VL-72B around 70.2 and Gemini 2.0 Flash around 71.9 in CodeSOTA’s benchmark table. Such registries are useful for trajectory tracking but should be treated as less authoritative than controlled paper evaluations because they often combine self-reported, vendor-reported, and independently verified numbers under varying protocols (CodeSOTA MMMU leaderboard). codesota.com
2025–2026: frontier models reach the 80% range, depending on protocol
OpenAI’s official GPT-5 system card-style announcement reports GPT-5 at 84.2% on MMMU under high-reasoning evaluation, describing it as a state-of-the-art multimodal-understanding result (OpenAI GPT-5 announcement). OpenAI
Current third-party leaderboards are not perfectly aligned. PricePerToken’s LayerLens-derived MMMU leaderboard, viewed on May 19, 2026, lists top MMMU results around 79.2 and reports 64 evaluated models, while LLM Stats’ MMMU validation page lists Claude Opus 4.5 at 80.7 across a much smaller set and explicitly marks those entries as self-reported. CodeSOTA shows still higher top entries, but it also aggregates across trust tiers and source types (PricePerToken MMMU leaderboard; LLM Stats MMMU validation; CodeSOTA MMMU leaderboard). Price Per Token+2LLM Stats+2
The safe conclusion is not “MMMU is solved.” The safe conclusion is narrower: frontier multimodal models have moved from the mid-50s to the 70%+ and sometimes 80%+ range on reported MMMU-style evaluations, but exact rankings depend heavily on split, prompt, reasoning budget, leaderboard governance, and whether the result is vendor-reported, third-party-reported, or independently reproduced.
Trajectory table
| Era / model family | Representative reported MMMU result | Interpretation |
|---|---|---|
| Random / frequent-choice baselines | Low-to-mid 20s | Establishes multiple-choice floor |
| GPT-4 text-only | ~34% | Strong text model cannot solve MMMU without visual input |
| Early open-source LMMs | Often 30s–40s | Large gap versus GPT-4V at launch |
| GPT-4V | ~56% | First strong frontier multimodal baseline, still far from human expert |
| Gemini Ultra / Claude 3 Opus | ~59% in MMMU comparisons | Slight improvement over GPT-4V-era baseline under reported protocols |
| GPT-4o / Claude 3.5 Sonnet / Gemini 1.5 Pro | High 60s in MMMU-Pro paper’s original-MMMU comparison | Stronger vision-language integration but still below upper human bands |
| Qwen2.5-VL / Gemini 2.x-class models | Low 70s in third-party registries | Open and commercial models cross the 70% threshold |
| GPT-5-class frontier models | Around 80%+ in official or third-party reports | Original MMMU may be less discriminative at the top, but protocol differences are large |
This table intentionally mixes only representative reported values, not a single canonical ranking. For modern MMMU results, leaderboard numbers should be read as evidence of a performance band, not as a clean ordering of model intelligence.
Methodological strengths
1. MMMU is genuinely multimodal
The strongest evidence that MMMU is not just a text benchmark with pictures is the gap between text-only and multimodal results. GPT-4 text scored in the mid-30s, while GPT-4V scored in the mid-50s. OCR and captioning did not erase that difference, which implies that the benchmark often requires joint interpretation of visual and textual information rather than post-hoc caption substitution (Yue et al. 2024 — CVPR). openaccess.thecvf.com
This matters for OCR as a Baseline. Many multimodal tests accidentally reward extracting visible text from an image and passing it to a language model. MMMU includes some text-bearing images, but its domain visuals often require interpreting structure: axes, spatial relationships, formal symbols, circuit logic, diagram conventions, image regions, or discipline-specific notation.
2. It targets expert-like academic tasks
MMMU’s college-level framing differentiates it from everyday VQA. A model that performs well on image captions, object recognition, or simple visual question answering may still fail at MMMU because it lacks field-specific visual literacy. The dataset’s 30-subject and 183-subfield design explicitly tries to sample across forms of expertise rather than across everyday scene categories (Yue et al. 2024 — CVPR). openaccess.thecvf.com
This makes MMMU a useful bridge between Multimodal Benchmarks and applied expert workflows. It is still a benchmark, not a professional certification test, but it asks questions closer to what students and practitioners actually do with technical images.
3. Heterogeneous image types reduce shortcut specialization
The paper emphasizes heterogeneous image types: charts, diagrams, maps, tables, medical images, chemical structures, music notation, design artifacts, and other visual forms. This diversity makes it harder for a model to specialize in one narrow visual channel and claim broad multimodal competence (Yue et al. 2024 — CVPR). openaccess.thecvf.com
Heterogeneity is especially important because multimodal models can be uneven. A system may be excellent at document screenshots and weak at spatial geometry, or good at charts and poor at medical scans. MMMU’s image-type breakdowns help expose that unevenness.
4. Expert-curated questions are harder to game than generic web VQA
MMMU’s collection process used domain-specialized students and coauthors rather than anonymous crowdworkers writing generic questions. The authors also attempted to avoid immediately searchable answers and filter duplicates. This does not eliminate contamination, but it raises the quality bar relative to many automatically collected datasets (Yue et al. 2024 — CVPR). openaccess.thecvf.com
The benchmark also includes difficulty labels and subject metadata, allowing analysis beyond one aggregate score. That makes MMMU more useful for model development: failure analysis can target domain, image type, or difficulty rather than merely “overall multimodal ability.”
Methodological weaknesses
1. Image quality variance is both realistic and confounding
MMMU’s images come from textbooks, exams, online resources, and newly created materials. That diversity is realistic, but it also introduces variance in resolution, cropping, formatting, visual clarity, labeling conventions, and source style. In a benchmark about expert visual reasoning, such variance can become either a valid part of the task or an uncontrolled nuisance factor.
This is not a simple flaw. Real expert settings often involve messy figures, screenshots, scans, or diagrams. But if one model is better at high-resolution document processing while another is better at reasoning after clean perception, a single MMMU score may conflate perception robustness with disciplinary reasoning. MMMU-Pro’s later use of screenshots, photos, varied backgrounds, font styles, and font sizes makes this issue explicit rather than accidental (Yue et al. 2025 — MMMU-Pro). arXiv
2. Multiple-choice format creates shortcuts
MMMU is about 94% multiple-choice. Multiple-choice evaluation is practical and reproducible, but it changes the task. A model can exploit answer-option priors, eliminate implausible options, or benefit from domain knowledge without fully solving the visual problem. The original paper’s answer-extraction rules and invalid-answer fallbacks also mean that instruction-following reliability can affect score (Yue et al. 2024 — CVPR). openaccess.thecvf.com
This is one reason MMMU-Pro expands the number of answer options and introduces vision-only variants. Increasing from four options to ten makes guessing and option-prior exploitation harder, and embedding more text into images forces the model to jointly see and reason rather than outsource the problem to the language channel.
3. Prompt and format dependence complicate leaderboards
MMMU scores can vary with prompt wording, chain-of-thought use, image resolution, model settings, and answer parsing. The MMMU paper used zero-shot evaluation and specific prompts, while later leaderboards often differ in system prompts, reasoning effort, and model versions. GPT-5’s official MMMU result, for example, is reported under high reasoning effort, which is not necessarily comparable to earlier zero-shot, fixed-prompt evaluations (OpenAI GPT-5 announcement). openaccess.thecvf.com
The consequence is that modern MMMU rankings are not as clean as they look. A reported 80% score may reflect a stronger model, a more favorable protocol, higher test-time reasoning budget, or all of the above. For Leaderboard Governance, MMMU is a case study in why benchmark reports need split, prompt, decoding, reasoning-budget, and verification metadata.
4. Contamination concerns have grown over time
The original authors were aware of contamination risk and tried to select questions whose answers were not immediately available online. That was a useful mitigation at construction time, but it cannot fully prevent leakage once benchmarks become influential. The MMMU GitHub repository states that test-set answers were released on February 12, 2026, enabling local evaluation but also weakening the hidden-test protection that supported earlier leaderboard credibility (Yue et al. 2024 — CVPR; MMMU GitHub). openaccess.thecvf.com
This does not invalidate MMMU. It changes how results should be interpreted. Post-release MMMU scores are still useful for reproducible evaluation, regression testing, and approximate comparison, but they are weaker evidence of uncontaminated generalization than pre-release hidden-test submissions.
5. The aggregate score hides domain-specific failure
A single MMMU score compresses many different abilities. A model might improve because it gets better at OCR-heavy diagrams, because it has more memorized domain knowledge, because it has stronger chart reasoning, or because it follows multiple-choice instructions better. The paper’s own error analysis shows that failures come from perception, knowledge, and reasoning, not one bottleneck (Yue et al. 2024 — CVPR). openaccess.thecvf.com
For engineering use, subject-level and image-type-level breakdowns are often more informative than the headline number. A medical-document assistant, a STEM tutor, and an engineering CAD assistant care about different subsets of MMMU competence.
Relationship to MMMU-Pro
MMMU-Pro is a harder derivative benchmark designed to address suspected shortcuts in the original MMMU. Its authors introduce a three-step process: filter out questions answerable by text-only models, expand answer options, and create a vision-only setting where question text and options are embedded into images. The paper reports that this produces substantial model-score drops, often in the range of 16.8 to 26.9 percentage points relative to original MMMU-style evaluation (Yue et al. 2025 — MMMU-Pro). ACL Anthology
MMMU-Pro contains 3,460 questions: 1,730 standard items and 1,730 screenshot/photo-style vision-only items. Human annotators manually created photo and screenshot variants with varied backgrounds, font styles, and font sizes, making the benchmark more robust to models that depend on clean textual channels (Yue et al. 2025 — MMMU-Pro). arXiv
The key design change is not merely “harder questions.” MMMU-Pro attacks three specific failure modes of the original benchmark:
| Original MMMU risk | MMMU-Pro intervention |
|---|---|
| Text-only solvability | Filter questions that text-only models can answer |
| Multiple-choice guessing | Expand answer options from four to ten |
| Language-channel shortcut | Embed question and options into images in a vision-only setting |
| Clean-format overfitting | Use screenshots/photos with varied presentation |
The MMMU-Pro paper’s table shows how scores fall under these transformations. GPT-4o, for example, is reported at 69.1 on original MMMU validation, 64.7 on MMMU-Pro standard four-option, 54.0 on standard ten-option, and 49.7 on the vision setting. Claude 3.5 Sonnet, Gemini 1.5 Pro, and Qwen2-VL-72B show similar declines (Yue et al. 2025 — MMMU-Pro). arXiv
MMMU-Pro also provides a useful diagnostic: OCR alone is not enough. The paper reports that adding OCR prompts did not significantly improve performance in the vision setting, and that high OCR accuracy did not guarantee correct reasoning. In one analysis of GPT-4o errors, reasoning mistakes were the dominant category, while text recognition was not the primary bottleneck (Yue et al. 2025 — MMMU-Pro). arXiv
In practical terms, MMMU-Pro is the benchmark to consult when the question is: “Are frontier models exploiting MMMU’s format?” MMMU remains useful for broad historical comparison; MMMU-Pro is more useful for stress-testing whether multimodal models can still solve the task when the easy textual and multiple-choice affordances are reduced.
Relationship to MathVista and ScienceQA
MMMU is part of a larger family of multimodal reasoning benchmarks, but its niche is distinct.
| Benchmark | Main focus | Typical level / scope | Relationship to MMMU |
|---|---|---|---|
| ScienceQA | Multimodal science question answering, often textbook-style | Mostly elementary and middle-school science in MMMU’s comparison | Earlier multimodal QA benchmark; less expert-level and less visually diverse than MMMU |
| MathVista | Mathematical reasoning in visual contexts | Visual math, charts, diagrams, geometry, quantitative reasoning | Complementary: narrower but deeper on math-visual reasoning |
| MMMU | Broad college-level multimodal academic reasoning | Six disciplines, 30 subjects, 183 subfields | Multimodal analogue of MMLU; broad expert-oriented coverage |
| MMMU-Pro | Robustified MMMU-style reasoning | Filtered, expanded-option, vision-only variants | Harder diagnostic for shortcut resistance |
ScienceQA is an important predecessor because it combines images and science questions, but MMMU’s own comparison notes that ScienceQA is less advanced in academic level and image-type coverage. MMMU reports ScienceQA as having about 6,000 examples, five image types, textbook sources, and multiple-choice format, while MMMU has 11,550 examples, 30 image types, interleaved inputs, and both multiple-choice and open-ended questions (Yue et al. 2024 — CVPR). openaccess.thecvf.com
MathVista occupies a different niche. Its official project description frames it as a benchmark for mathematical reasoning in visual contexts, with 6,141 examples derived from 28 existing multimodal datasets and three newly created datasets: IQTest, FunctionQA, and PaperQA. It is therefore a stronger comparator when the question is visual mathematical reasoning, while MMMU is the stronger comparator when the question is broad disciplinary multimodal competence (MathVista project site). MathVista
The practical distinction is this: ScienceQA tests earlier-stage multimodal science QA; MathVista tests math-heavy visual reasoning; MMMU tests broad college-level multimodal reasoning; MMMU-Pro tests whether MMMU-style reasoning survives reduced shortcuts.
Has MMMU saturated?
The answer depends on what “saturated” means.
If saturation means “frontier models are no longer near the original GPT-4V range,” then yes: the benchmark has clearly moved. GPT-4V’s roughly 56% result is no longer frontier. Reported GPT-4o/Claude/Gemini/Qwen-class systems crossed the high-60s and low-70s, and GPT-5-style frontier reports reach the 80% range under some protocols (Yue et al. 2024 — CVPR; OpenAI GPT-5 announcement; CodeSOTA MMMU leaderboard). codesota.com+3openaccess.thecvf.com+3arXiv+3
If saturation means “the benchmark no longer distinguishes any models,” then no. Third-party leaderboards still show a broad distribution across models, and MMMU-Pro reveals large drops when shortcut-resistant transformations are applied. For non-frontier and mid-frontier models, MMMU remains discriminative.
If saturation means “the aggregate original-MMMU score may be losing resolution among the best multimodal systems,” then probably yes, but the evidence is not clean enough for a definitive claim. Several signs point in that direction: frontier systems cluster in the 70%–80%+ range, human lower-bound performance is no longer wildly distant from the best reported systems, answer release increases contamination risk, and protocol differences create noise large enough to obscure small model gaps. But no single, current, independently governed hidden-test reevaluation of all frontier models under one protocol establishes that the benchmark is fully saturated.
The most intellectually honest position is:
MMMU remains historically important and still useful, but the original aggregate score is no longer sufficient as a frontier multimodal discriminator by itself. For frontier evaluation, it should be paired with MMMU-Pro, per-subject breakdowns, per-image-type breakdowns, contamination-aware splits, and newer hidden or dynamically generated multimodal tasks.
This is the same pattern seen in many Benchmark Saturation cycles. A benchmark first exposes a capability gap, then drives model progress, then becomes a coarse regression test, and finally needs harder variants or better-governed hidden evaluations to remain frontier-sensitive.
Practical interpretation for AI engineering
For model developers, MMMU is best used as a broad diagnostic, not as a single optimization target. A rising MMMU score suggests improved multimodal academic reasoning, but it does not identify whether the improvement came from better image encoders, stronger OCR, better instruction following, larger language-model priors, more test-time reasoning, or contamination.
A rigorous evaluation stack should treat MMMU as one layer:
| Evaluation question | Recommended evidence |
|---|---|
| Does the model handle broad college-level multimodal tasks? | MMMU aggregate score |
| Which disciplines are weak? | MMMU subject-level breakdown |
| Which visual forms are weak? | MMMU image-type breakdown |
| Is the model exploiting text-only shortcuts? | MMMU-Pro filtered and vision-only settings |
| Is the model strong at visual math? | MathVista |
| Is the model strong at simpler science QA? | ScienceQA |
| Is the result robust? | Repeated runs, fixed prompts, fixed resolution, independent scoring, hidden or fresh items |
For Self-Improving Systems, MMMU is especially tempting as an automated reward or selection signal. That should be done carefully. Optimizing a model against MMMU alone risks teaching it to exploit multiple-choice structure, benchmark-specific priors, and known examples. A better use is as one held-out diagnostic in a suite that includes harder private tasks, adversarial variants, domain-specific tests, and qualitative failure analysis.
References
Companion entries
Core theory: MMLU, Multimodal Reasoning, Expert AGI, Vision-Language Models, Benchmark Saturation
Benchmark family: MMMU-Pro, MathVista, ScienceQA, MM-Vet, ChartQA, AI2D, VQA Benchmarks
Evaluation practice: Leaderboard Governance, Prompt Sensitivity, OCR as a Baseline, Hidden Test Sets, Contamination-Aware Evaluation, Per-Task Error Analysis
Model development: Frontier Multimodal Models, Test-Time Reasoning, Self-Improving Systems, Synthetic Evaluation Data, Domain-Specific Model Evaluation
Counterarguments: Multiple-Choice Evaluation Limits, Benchmark Gaming, Data Contamination, Human Expert Baselines, Goodhart’s Law in AI Evaluation