OSWorld is the benchmark that made “computer use” mean more than browser automation: it evaluates agents inside real, virtualized operating-system environments where success depends on seeing the screen, clicking, typing, using desktop applications, manipulating files, and coordinating multi-application workflows. Its central contribution is architectural as much as statistical: it turns the desktop operating system into the evaluation environment for Computer-Use Agents, exposing failures that web-only and terminal-only benchmarks cannot see. arXiv
Coverage note: verified through May 12, 2026.
Why OSWorld matters
OSWorld was introduced by Xie et al. in 2024 as “a scalable, real computer environment” for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across operating systems including Ubuntu, Windows, and macOS. The published benchmark contains 369 real-world computer tasks involving web and desktop applications, OS file I/O, and workflows spanning multiple applications; the project page also notes an additional 43 Windows tasks used for analysis. arXiv
The benchmark arrived at a moment when the field was starting to distinguish between ordinary tool use and GUI Grounding: not merely calling APIs, but operating the same interfaces humans operate. OSWorld’s initial result was deliberately sobering. Humans completed 72.36% of tasks, while the best evaluated model achieved 12.24%, with failures concentrated around grounding, procedural knowledge, and operating-system interaction rather than abstract language reasoning alone. arXiv
That gap made OSWorld a reference point for a new class of agent systems. Anthropic’s computer-use release evaluated Claude 3.5 Sonnet on OSWorld; OpenAI’s Computer-Using Agent, powering Operator, reported OSWorld alongside WebArena and WebVoyager; Google’s Gemini 2.5 Computer Use model card included an OSWorld row even while marking desktop OS control as unsupported. Anthropic Brand Portal+2OpenAI+2
The resulting story is not simply “scores went up.” It is that OSWorld forced vendors and researchers to answer a harder question: can an AI system operate a full computer, not just a website, terminal, or API?
What OSWorld evaluates
OSWorld evaluates agents that interact with a live desktop environment through observations and actions. In the common setting, the agent receives screenshots, reasons about the current state, and emits actions corresponding to mouse, keyboard, and GUI operations. This matches how OpenAI describes its Computer-Using Agent loop: screenshots enter the model context, the model reasons over current and past states, and actions such as clicking, scrolling, and typing are executed through a virtual mouse and keyboard. OpenAI
The benchmark tasks are meant to approximate ordinary desktop work rather than puzzle-box interaction. They include office productivity, web browsing, desktop application use, file manipulation, system-level tasks, and cross-application workflows. Anthropic’s October 2024 model card characterized OSWorld as measuring tasks involving web and desktop applications, OS file I/O, and multi-application workflows, using screenshot-only inputs in its reported evaluation even though OSWorld can expose other inputs such as accessibility-tree text. Anthropic Brand Portal
OSWorld’s infrastructure is organized around task initialization, interaction, post-processing, file retrieval, and execution-based evaluation. The project page describes configuration files that initialize tasks, mediate agent interaction, perform post-processing, retrieve files or state, and execute evaluation functions; it also supports parallel and headless execution. OSWorld
The important design choice is that success is not measured by imitation of a reference trajectory. A task can be solved through any valid sequence of GUI actions as long as the resulting state satisfies the evaluator. This is the same philosophical move behind Outcome-Based Agent Evaluation: evaluate the state of the world after the agent acts, not whether it followed the same path as a human demonstration.
The architectural contribution: a full operating-system benchmark
OSWorld’s core contribution is not another task list. It is the decision to make the operating system itself the environment.
Earlier digital-agent benchmarks often targeted one surface: static screenshots, mobile UIs, web navigation, toy web widgets, code interpreters, or command-line tasks. OSWorld instead places the agent inside a general-purpose computer environment. This matters because desktop work is compositional. A mundane task may require opening a file manager, downloading a file, editing a document in LibreOffice, switching windows, using browser search, saving an output artifact, and returning to a folder. A web-only benchmark can test navigation and form filling, but it cannot fully test this cross-application operating context. A terminal-only benchmark can test shell competence, but it cannot test visual UI grounding or mouse-driven application workflows.
The benchmark comparison on the OSWorld project page makes this explicit. OSWorld is marked as a controllable executable “Computer” environment with multimodal support, cross-application tasks, intermediate initial states, and 134 execution-based evaluation functions. OSWorld
That places OSWorld near the center of Embodied Software Agents research. It is “embodied” not in a robot body, but in a software body: the agent must act in an environment where its observations are partial, actions have side effects, and recovery from mistakes is part of the task. A model that can answer “how do I change line spacing in LibreOffice?” may still fail if it cannot locate the menu, select the right paragraphs, avoid disturbing the document, and save the final file.
This architectural shift also changes what “generalization” means. In OSWorld, generalization is not only about unseen instructions; it includes unseen UI layouts, timing delays, window focus, application-specific conventions, OS-specific keyboard shortcuts, and file-system side effects. These are mundane for humans and brutal for agents.
Task and evaluation pipeline
A typical OSWorld evaluation proceeds through four layers.
First, the environment is initialized. The benchmark may set up files, browser state, documents, application windows, or system configuration. This is why intermediate initial states matter: the task need not begin from a blank desktop, and the evaluator can place the agent in the middle of a realistic workflow. OSWorld
Second, the agent observes the computer. In screenshot-only settings, the model receives visual frames and must infer UI state from pixels. Some configurations can expose auxiliary information, but the most important public comparisons emphasize screen-based interaction because it more closely resembles human visual computer use. Anthropic’s October 2024 OSWorld evaluation, for example, provided only screenshot inputs. Anthropic Brand Portal
Third, the agent acts. Actions are usually translated into GUI commands: click, type, drag, scroll, press a key combination, or otherwise operate the virtual machine. OSWorld’s GitHub examples show local and parallel evaluation with screenshot observation, headless execution, maximum step budgets, and result directories containing screenshots, actions, and video recordings. GitHub
Fourth, the benchmark evaluates the final state. Evaluation may inspect files, document contents, rendered output, browser state, application state, or other artifacts. This is why OSWorld is stronger than transcript-based scoring: it can accept many possible action paths if the endpoint is correct. The OSWorld project describes custom execution-based evaluation scripts for reliable and reproducible evaluation. arXiv
The pipeline is powerful but expensive. It requires maintaining working operating-system images, application versions, task assets, credentials, websites, timing assumptions, and evaluator code. OSWorld-Verified exists largely because this maintenance burden proved larger than expected.
Empirical trajectory: from failure to near-human claims
OSWorld’s empirical history has moved quickly. The initial 2024 paper showed frontier agents far below human performance. By 2025, specialized computer-use agents, agentic frameworks, and reasoning models had pushed scores dramatically upward. By 2026, vendor-reported OSWorld-Verified scores for frontier systems were near or above the original human baseline, though direct comparison requires care because OSWorld-Verified changed infrastructure, task quality, scoring, and sometimes evaluation settings. XLANG Lab+2Anthropic+2
| Period / source | Agent or system | Reported OSWorld result | What the number means |
|---|---|---|---|
| Xie et al. 2024 | Best evaluated baseline | 12.24% | Original OSWorld benchmark; humans at 72.36%; failures attributed mainly to GUI grounding and operational knowledge. arXiv |
| Xie et al. 2024 | Human baseline | 72.36% | Human success rate on the original benchmark, not an upper bound on expert or repeated-attempt performance. arXiv |
| Anthropic, Oct. 2024 | Claude 3.5 Sonnet computer use | 14.9% at 15 steps; 22% at 50 steps | Screenshot-only OSWorld evaluation; Anthropic explicitly noted that allowing more steps improved success. Anthropic Brand Portal |
| OpenAI, Jan. 2025 | Computer-Using Agent / Operator | 38.1% | OpenAI reported CUA at 38.1% on OSWorld and described it as still unreliable for OS automation without human oversight. OpenAI |
| OSWorld-Verified, July 2025 | CoACT-1 | 60.76% | Verified rerun result; OSWorld team said agentic frameworks powered by reasoning models were leading the leaderboard. XLANG Lab |
| OSWorld-Verified, July 2025 | Agent S2.5 w/ o3; GTA1 w/ o3 | 56.0%; 53.1% | Strong results from orchestrated agentic frameworks, not just raw base-model prompting. XLANG Lab |
| OSWorld-Verified, July 2025 | Claude 4 Sonnet; UI-TARS | 43.9%; 40.0% | OSWorld-Verified reported Claude 4 Sonnet approaching specialized computer-use models such as UI-TARS. XLANG Lab |
| Google, Oct. 2025 | Gemini 2.5 Computer Use / Mariner-related systems | No OSWorld score; OS control not yet supported | Google’s model card included OSWorld but marked Gemini’s desktop OS control as unsupported; the model was optimized primarily for web browsers and showed mobile promise. blog.google |
| Anthropic, Feb. 2026 | Claude Sonnet 4.6 / Opus 4.6 | 72.5%; 72.7% | Anthropic’s Sonnet 4.6 system-card snippet reports OSWorld-Verified at 72.5%, within 0.2% of Opus 4.6 at 72.7%, using 1080p and a 100-step limit. Anthropic |
| OpenAI, Apr. 2026 | GPT-5.5 | 78.7% | OpenAI’s GPT-5.5 release reports OSWorld-Verified at 78.7%, with GPT-5.4 at 75.0% and Claude Opus 4.7 at 78.0% in the same table. OpenAI |
The trend is real, but the table should not be read as a clean longitudinal curve. The original OSWorld score, vendor self-reports, OSWorld-Verified reruns, different step budgets, different screen resolutions, different action abstractions, and different orchestration layers all affect the result. Anthropic itself notes that scores before Claude Sonnet 4.5 were measured on original OSWorld, while scores from Sonnet 4.5 onward use OSWorld-Verified, which updates task quality, grading, and infrastructure. anthropic.com
The practitioner lesson is that an OSWorld score is not a scalar property of a model alone. It is a property of a model, prompt, harness, observation channel, action space, step budget, runtime, evaluator version, and sometimes an orchestration framework. Treat it like a systems benchmark, not like a multiple-choice exam.
Anthropic Computer Use
Anthropic’s October 2024 “computer use” release framed the capability as a model interpreting screenshots and generating GUI commands to navigate websites, click, type, and complete multi-step tasks. Its model card addendum evaluated Claude 3.5 Sonnet on OSWorld and reported 14.9% success with the standard 15-step limit and 22% when increasing the limit to 50 steps. anthropic.com
This result is important for two reasons. First, it showed that frontier VLMs could begin to operate real GUIs but were still brittle. Second, it showed that evaluation settings such as step budget materially affect success. If a model fails because it runs out of steps while otherwise navigating correctly, the benchmark is measuring planning efficiency as well as task competence. That is legitimate, but it must be reported.
By 2026, Anthropic’s own materials described OSWorld as a standard benchmark for computer use, involving real software such as Chrome, LibreOffice, and VS Code running on a simulated computer with no special APIs or connectors. Anthropic also distinguished original OSWorld from OSWorld-Verified, emphasizing that newer scores use the verified version. anthropic.com
OpenAI Operator and Computer-Using Agent
OpenAI’s Operator was powered by Computer-Using Agent, a model combining GPT-4o vision with reinforcement-learning-enhanced reasoning and trained to interact with GUIs through the same visible interface humans use. OpenAI reported 38.1% on OSWorld, 58.1% on WebArena, and 87.0% on WebVoyager, describing CUA as using a universal interface of screen, mouse, and keyboard. OpenAI
OpenAI’s Operator system card is also useful because it treats OSWorld not merely as a capability benchmark but as a risk signal. The card says CUA’s 38.1% OSWorld result indicates it was not yet highly reliable for automating OS tasks and recommends human oversight, especially in non-browser environments. OpenAI
This is exactly how OSWorld should be used in deployment decisions. A high score indicates competence on a task distribution; a low or moderate score indicates that unsupervised desktop automation will produce mistakes. It does not certify safety, policy compliance, data handling, or prompt-injection resistance.
OpenAI’s later GPT-5.5 release reports a much higher OSWorld-Verified number, 78.7%. That number is notable because it crosses the original human baseline, but it is a vendor-reported score in a later benchmark variant and should be interpreted alongside details about harness, action budget, model effort, and evaluation environment. OpenAI
Google Mariner and Gemini Computer Use
Google’s computer-use story is different. Gemini 2.5 Computer Use, used in systems including Project Mariner-related experiences, was explicitly optimized for web browsers and showed promise on mobile UI control, but Google stated that it was not yet optimized for desktop OS-level control. blog.google
That distinction matters because the user-visible phrase “computer use” can hide incompatible evaluation surfaces. Google’s model card reports strong web and mobile benchmark results, including Online-Mind2Web, WebVoyager, and AndroidWorld, but its OSWorld row says “OS control not yet supported.” In the same table, OpenAI CUA is listed at 38.1%, Claude Sonnet 4 at 42.2%, and Claude Sonnet 4.5 at 61.4% for OSWorld. Google Cloud Storage
So the correct comparison is not “Mariner scored worse on OSWorld.” The correct comparison is: Google’s released computer-use model targeted web and mobile surfaces, not full desktop OS operation, while OSWorld measures the latter. This is a concrete example of Benchmark Fragmentation by task class.
Methodology strengths
Genuine OS-level tasks
OSWorld’s strongest feature is that the agent acts in a real desktop environment rather than a simplified simulator. The published tasks involve real web and desktop applications, file I/O, and workflows spanning multiple applications. arXiv
This matters because many agent failures live in the interface layer. The model may know the answer but fail to select the right menu. It may use the wrong keyboard shortcut for the OS. It may click the wrong target because the UI is shifted by a toolbar. It may save the file in the wrong directory. OSWorld makes these failures visible.
Execution-based evaluation
OSWorld’s use of custom execution-based evaluators is a major methodological advantage. In principle, it permits multiple valid solution paths and scores the resulting state rather than a prescribed action trace. The project page identifies 134 execution-based evaluation functions. OSWorld
This is crucial for open-ended desktop work. A human can resize an image, format a document, or move a file through several different paths. A benchmark that only accepts one trajectory would punish valid competence.
Cross-application composition
Many useful desktop workflows are multi-application. OSWorld’s design includes cross-app tasks and intermediate initial states, which allows it to evaluate task continuation rather than only task initiation. OSWorld
This makes it more relevant to agentic work than one-shot UI classification datasets. A realistic agent must track goals through a sequence of application state changes.
Open infrastructure and public evaluation
The OSWorld repository supports local evaluation and public verified evaluation. For verified leaderboard results, the maintainers require running agent code on their side or sharing monitoring data and trajectories, and they ask submitters to disclose enough implementation detail for the public to understand what happened. GitHub
That is important because computer-use benchmarks are unusually easy to make incomparable. Small differences in VM images, timing, browser state, display resolution, and action wrappers can change success rates.
Methodology weaknesses
Infrastructure noise
OSWorld’s realism creates noise. The OSWorld-Verified post is unusually candid about this. The team reports that after the first release, despite substantial checking and maintenance, they collected roughly 300 issues from several institutions. Problems included CAPTCHAs, IP blocking, network restrictions, dynamic website changes, URL evolution, timing dependencies, and setup failures. XLANG Lab
This is not a minor annoyance. It means a benchmark task can fail because an external website changed, a page loaded slowly, a bot-detection system fired, or a file-hosting provider throttled downloads. The OSWorld project page even notes that 8 Google Drive-related tasks may require manual configuration or be excluded, yielding a permitted 361-task evaluation. OSWorld
OSWorld-Verified addressed many of these problems by migrating infrastructure toward AWS, improving parallelization, moving files from Google Drive to Hugging Face, and repairing task/evaluator issues. XLANG Lab
Scoring brittleness
Execution-based scoring is better than trajectory matching, but it can still be brittle. The OSWorld-Verified post lists examples: ambiguous color descriptions, unclear spreadsheet ranges, GIMP resize ambiguity, multiple valid paragraph-formatting interpretations, and functionally equivalent spreadsheet formulas being marked wrong. XLANG Lab
The repair strategy included clarifying instructions, adding acceptable alternatives, fuzzy matching, perceptual hashing for visual comparisons, and more tolerant document/PDF checks. XLANG Lab
This reveals a broader lesson for Agent Evaluation Harnesses: verifiers are software, and software encodes assumptions. A benchmark can be too strict, too loose, or silently wrong.
OS-specific quirks
OSWorld’s operating-system realism creates OS-specific artifacts. Desktop resolution, window manager behavior, file-dialog ordering, keyboard shortcuts, installed application versions, fonts, locale, browser state, and accessibility APIs can all matter. A model may perform well on Ubuntu but fail on Windows or macOS because the UI conventions differ. The OSWorld project explicitly notes Ubuntu, Windows, and macOS support, with additional Windows tasks for analysis. OSWorld
This is not a flaw so much as a measurement choice. Real users also face OS-specific interfaces. But practitioners should not treat an OSWorld score as proof that an agent will work equally well across every desktop environment.
Task-distribution limits
OSWorld covers office, productivity, web, file, and system workflows, but it does not cover all professional computer use. The OSWorld-Verified team says the first iteration selected commonly used software and avoided tasks requiring extended completion times, deep professional software expertise, video understanding, real-time intensive operations, and asynchronous completion scenarios. XLANG Lab
This is a major limitation for production claims. A high OSWorld score does not imply mastery of Photoshop, AutoCAD, Bloomberg Terminal, CAD/CAM software, advanced spreadsheet modeling, IDE debugging, or enterprise SaaS workflows unless those distributions are separately tested.
Step budgets and latency
Computer-use success depends on patience. Anthropic’s Claude 3.5 Sonnet score rose from 14.9% at 15 steps to 22% at 50 steps. Anthropic Brand Portal
That is not just a benchmark technicality. It reflects a real product constraint: users care about speed, cost, and recoverability. A system that can complete a task in 120 GUI actions may be impressive as a research artifact but unusable as a coworker if it takes many minutes, consumes excessive tokens, or fails silently halfway through.
Training data and scaffold dependence
The OSWorld-Verified team observed that current leading models’ success often stems from extensive human trajectory data, and that agentic frameworks powered by reasoning models were leading the verified leaderboard. XLANG Lab
This complicates interpretation. A score may reflect the base model, a GUI-specialized model, a planner-grounder architecture, a test-time scaling strategy, or a large corpus of demonstrations. For research, that is fine if disclosed. For model comparison, it is fatal if hidden.
OSWorld, WebArena, and Terminal-Bench
OSWorld is best understood as part of a three-surface evaluation map: desktop GUI, web GUI, and terminal.
| Benchmark | Primary surface | What it tests well | What it misses |
|---|---|---|---|
| OSWorld | Full desktop operating system | Visual GUI grounding, office/productivity tasks, desktop apps, file I/O, cross-application workflows, OS operations | Deep professional verticals, very long asynchronous work, highly controlled web reproducibility, terminal-only depth |
| WebArena | Self-hosted web applications | Browser navigation, web forms, long-horizon web tasks, reproducible web environments, programmatic validation | Desktop apps, OS file I/O, non-browser GUI work |
| Terminal-Bench | Command-line environments | Shell use, coding, server setup, build/test workflows, text-based automation, deterministic test scripts | Visual GUI grounding, desktop applications, mouse/keyboard UI operation |
WebArena is a standalone, self-hostable web environment with websites modeled on real-world equivalents and validators for functional correctness. Its paper reports 812 long-horizon web tasks and an early best GPT-4 agent success rate of 14.41% versus human performance of 78.24%. WebArena
Terminal-Bench evaluates agents in real terminal environments, including tasks such as compiling code, training models, and setting up servers. Its repository describes a task suite plus an execution harness connecting language models to a sandboxed terminal, while the Terminal-Bench 2.0 paper describes 89 hard terminal tasks with unique environments, human-written solutions, and comprehensive tests. GitHub
The relationship is therefore complementary, not hierarchical. WebArena is not obsolete because OSWorld exists; it is more focused and often more reproducible for browser agents. Terminal-Bench is not weaker because it lacks pixels; it is the right benchmark for command-line autonomy. OSWorld is the right benchmark when the claim is full computer use.
Which environment should practitioners evaluate against?
The practical question is not “which benchmark is best?” The question is “what interface will the deployed agent actually control?”
| Product claim | Primary benchmark choice | Secondary checks |
|---|---|---|
| “Our agent can use desktop software like a human.” | OSWorld / OSWorld-Verified | Internal desktop-app tasks, OS-specific regression suites, human trajectory review |
| “Our agent can browse websites and complete web workflows.” | WebArena, WebVoyager, Online-Mind2Web | Live-site robustness tests, prompt-injection tests, login/session handling |
| “Our agent can operate developer terminals and servers.” | Terminal-Bench | SWE-bench-style coding tasks, internal DevOps sandboxes, security constraints |
| “Our agent can use mobile apps.” | AndroidWorld or similar mobile benchmarks | Device-specific UI tests, latency and recovery tests |
| “Our agent can automate enterprise workflows.” | Internal task suite first | OSWorld/WebArena/Terminal-Bench only as external calibration |
| “Our model is generally agentic.” | Multiple surfaces | Report scores separately; do not average away surface-specific failure modes |
For a desktop assistant, OSWorld should be a default external benchmark, but not the only evaluation. A production agent needs additional tests for the exact applications, file types, permissions, business rules, and failure consequences in the target environment.
For a browser agent, OSWorld can be useful as a stress test, but WebArena-style benchmarks are more directly relevant. Google’s Gemini 2.5 Computer Use model is the clearest example: it reports strong web and mobile control results, while desktop OS control is not yet supported. blog.google
For a terminal or coding agent, Terminal-Bench is usually more relevant than OSWorld because it tests the command-line action space directly. OpenAI’s GPT-5.5 release, for example, reports both Terminal-Bench 2.0 and OSWorld-Verified, implicitly treating them as different dimensions of agentic capability. OpenAI
A mature evaluation stack should report at least five details: benchmark version, environment version, observation channel, action space, and step/time budget. Without these, OSWorld comparisons become marketing numbers rather than engineering evidence.
Is OSWorld becoming the dominant benchmark?
Within full desktop computer use, yes: OSWorld is the default reference point. Anthropic, OpenAI, Google, OSWorld’s own verified leaderboard, and several specialized GUI-agent projects use it as the shared measurement surface. The OSWorld-Verified update explicitly describes the benchmark’s influence on the community and its adoption by major AI labs. XLANG Lab
But across all computer-use agents, the field is fragmenting by task class. Google’s computer-use model optimized for web and mobile rather than desktop OS control; WebArena remains the reference for self-hosted web tasks; Terminal-Bench is becoming a reference for terminal autonomy; mobile benchmarks cover a different action space; and enterprise agents increasingly need private workflow suites. blog.google+2arXiv+2
The likely equilibrium is not a single universal benchmark. It is a benchmark portfolio:
-
OSWorld for full desktop GUI and operating-system interaction.
-
WebArena and related benchmarks for web agents.
-
Terminal-Bench for shell and infrastructure agents.
-
AndroidWorld-like suites for mobile agents.
-
Domain-specific private benchmarks for enterprise deployment.
OSWorld may still become the headline score for “can this model use a computer?” But that headline will be misleading unless readers understand that “computer” is an interface family, not one task distribution.
What OSWorld reveals about computer-use agents
OSWorld makes three facts difficult to ignore.
First, language models are not automatically computer users. Knowing what to do is not the same as doing it through a noisy GUI. OSWorld’s initial 12.24% best-model result made that gap visible. arXiv
Second, computer-use ability is systems ability. The best results come from the combination of model, grounding, planning, memory, action representation, verification, recovery, and task-specific data. OSWorld-Verified’s leaderboard analysis emphasizes agentic frameworks, reasoning models, and trajectory data rather than treating success as a pure foundation-model property. XLANG Lab
Third, evaluation infrastructure is part of the science. OSWorld-Verified’s 300+ fixes, infrastructure migration, evaluator improvements, and public verification process show that a benchmark for live computer use is not a static dataset. It is an evolving measurement instrument. XLANG Lab
That last point may be OSWorld’s most important contribution. It demonstrates that benchmarking agents in realistic environments requires the same discipline as building production systems: observability, regression testing, version control, task health monitoring, and public reproducibility norms.
Cited primary sources
The core OSWorld source is Xie et al. 2024, “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments,” together with the OSWorld project page and repository documentation. arXiv+2OSWorld+2
The OSWorld-Verified source is XLANG Lab’s July 2025 post, which documents the infrastructure migration, task repairs, leaderboard reruns, and known benchmark limitations. XLANG Lab+3XLANG Lab+3XLANG Lab+3
The Anthropic sources are the October 2024 computer-use release, the Claude 3 model-card October addendum, and later Claude 4.6 materials describing OSWorld and OSWorld-Verified score conventions. Anthropic+3anthropic.com+3Anthropic Brand Portal+3
The OpenAI sources are the Computer-Using Agent release, the Operator system card, and the GPT-5.5 release table reporting OSWorld-Verified results. OpenAI+2OpenAI+2
The Google sources are the Gemini 2.5 Computer Use announcement and model card, which report web/mobile benchmark results and mark OSWorld desktop control as unsupported. blog.google
The comparison benchmarks are WebArena and Terminal-Bench. WebArena is a self-hostable web benchmark with programmatic validators; Terminal-Bench is a terminal-environment benchmark with task-specific tests and sandboxed execution. arXiv+3WebArena+3arXiv+3
Companion entries
Core theory: Computer-Use Agents, GUI Grounding, Embodied Software Agents, Outcome-Based Agent Evaluation, Verification Asymmetry
Benchmarks: OSWorld-Verified, WebArena, Terminal-Bench, AndroidWorld, WebVoyager, Windows Agent Arena
Practice: Benchmark Selection for AI Agents, Agent Evaluation Harnesses, Desktop Automation Agents, Human-in-the-Loop Computer Use, Trajectory Data for GUI Agents
Failure modes and counterarguments: Benchmark Brittleness, Infrastructure Noise in Agent Evaluation, Prompt Injection Against Agents, OS-Specific GUI Quirks, Reward Hacking in Agent Benchmarks, Benchmark Fragmentation