Computer Use: the GUI as an Action Space

The harness of Chapter 41 dispatches tools, and a tool, so far, has meant a machine interface: an API, a shell, a database. Most of the world's software offers no such interface. It offers a screen, built for human eyes and hands, and the long tail of enterprise systems, desktop applications, and vendor portals will never ship anything else. Computer use is the decision to treat that screen as the action space: the model receives screenshots, and acts by clicking coordinates and typing keys, operating software the way its users do. This chapter works through the design fork the field has carried for a decade, why locating a button is the bottleneck rather than deciding to click it, what it costs to give every session a live operating system, why the screen is the harshest prompt-injection surface in the book, and whether this whole interface is a permanent layer or scaffolding for a web that grows machine interfaces. By the end, a reader can explain why an agent that reasons impressively still misses a Submit button, and price the difference between a tool call and a GUI action.

The interface that is always there

The case for pixels was stated plainly at the technique's public debut: rather than building a tool per task, teach the model "general computer skills," so it can use "a wide range of standard tools and software programs designed for people" (Anthropic 2024). The alternative reading of that sentence is economic. Every API integration is built and maintained by someone; the screen is the one interface whose integration cost is already paid, by the application's own developers, for its human users.

The idea has an industrial ancestor whose scars are instructive. Robotic process automation built a real industry on scripted GUI operation: bots that locate elements by selectors and replay recorded action sequences against software that was never going to get an API. Its standing weakness is brittleness, a selector matches until a UI update breaks it, and industry lore prices bot maintenance as a substantial share of what the automation saves. A model-driven computer-use agent is RPA's problem statement with the perception layer replaced: instead of a hand-written selector, a vision-language model that finds the button the way a person does, and so, in principle, keeps finding it after the redesign. Whether the learned layer is reliable enough to cash that promise is this chapter's empirical question.

Pixels or the tree

Two representations of a screen compete for the observation and action space, and the fork is as old as the field: the 2017 environment that started the research line already exposed both (Shi et al. 2017).

The pixel path gives the model screenshots and asks it to emit coordinates. It is universal, a canvas app, a legacy terminal emulator, and a video frame all render the same way, and it matches how the model's vision encoder saw the world in pretraining. Its failure mode is the mis-grounded click: the model knows what to act on and misses where. The structured path reads the machine's own description of the interface, the DOM in a browser or the operating system's accessibility tree, and lets the model act on named elements with no pixel arithmetic at all. Its failure modes are the tree's: custom widgets and canvas-rendered interfaces are missing from it, it can lag the rendered state, and a full dump of a busy screen runs to thousands of tokens against one image. In between sits the workaround that carried the field's early agents: set-of-marks prompting overlays numbered tags on detected elements so the model outputs a tag rather than a coordinate (Yang et al. 2023). The empirical answer, measured since the first real-OS benchmark, is that hybrids win: screenshot plus tree outperforms either representation alone (Xie et al. 2024). The screen says how things look; the tree says what they are; the agent needs both.

Grounding is the bottleneck

Grounding is the step from "the thing I want to act on" to screen coordinates, and it is worth separating from planning, because the two fail independently and the field's surprise was which one dominates. Models plan competently on screens they cannot ground: the benchmark built from professional software, CAD tools, scientific applications, high-resolution screens with tiny targets, measured the best model of its moment at 18.9% grounding accuracy, while a training-free trick, iteratively zooming into candidate regions before predicting, nearly tripled that (Li et al. 2025). Perception, not reasoning, was the binding constraint, and that explains an otherwise odd product detail: mid-2026 computer-use APIs ship zoom actions, because professional interfaces at native resolution fall below what a downsampled screenshot can resolve.

Why a per-step perception error is fatal, rather than merely annoying, is one line of arithmetic: a task of $n$ actions with per-step success $p$ completes at roughly $p^n$. The runnable prices it. At 30 steps, the difference between 95% and 99% per step is the difference between a demo and a product, and real agents make it worse by taking two to four times more steps than a skilled human's trajectory needs (Abhyankar et al. 2025). Long-horizon GUI work compounds exactly the way Chapter 90's task-level reliability analysis says it must; grounding is just the term in the product that this chapter's hardware and perception limits set.

# end-to-end success ~= per-step success ^ steps (independence approximation)
steps = [5, 10, 30, 60, 100]
accs  = [0.90, 0.95, 0.99, 0.999]
print("steps " + "".join(f"   p={p:<5}" for p in accs))
for n in steps:
    print(f"{n:5d} " + "".join(f"  {p**n:6.1%} " for p in accs))
import math
p = 0.99
print(f"horizon at 50% success for p={p}: {math.log(0.5)/math.log(p):.0f} steps")
print("agents also take 2.7-4.3x more steps than the human-optimal path,")
print("which multiplies the exponent, not the base")

Two years up the benchmark

The capability's trajectory is unusually legible because one benchmark anchored it. OSWorld put agents in front of real operating systems, 369 tasks across actual applications with execution-based checking, and opened with a gap that defined the problem: humans above 72%, the best agent at 12.24% (Xie et al. 2024). The climb since reads as a compressed history of the field's methods. The first frontier-model beta, in late 2024, reached the low twenties while its own release notes called it cumbersome and error-prone (Anthropic 2024). A purpose-trained model with a hosted browser roughly doubled that within months (OpenAI 2025). The open end-to-end line, trained on large GUI corpora and then on reinforcement learning through hundreds of live virtual machines, doubled its own numbers year over year (Qin et al. 2025). By late 2025 frontier models passed 60%; by mid-2026, after the benchmark itself was repaired once and its harness revised again, reported scores sit around the original human baseline, with the vendor's own framing conceding they still trail skilled humans. Two readings of that curve coexist. One is genuine, steep capability growth, the same verifier-driven training the reasoning chapters describe, applied to trajectories of clicks. The other is a measurement caution this book's Chapter 47 installed: the ruler was revised twice mid-climb, and a score "at the human baseline" on 369 checkable tasks is not the claim that an agent replaces a human at a desk.

What a session costs

A tool call in Chapter 41 is a stateless request measured in milliseconds. A computer-use step is a different object: capture a screenshot, upload it as a thousand-plus image tokens with the accumulated history, run a frontier forward pass to plan and ground, execute the action, wait for the interface to settle, and do it again, seconds per step, tens of steps per task. Under the agent there must be a live, isolated operating system for the session's whole duration; the reference deployment is a container running a virtual display server with a desktop and browser inside, and a market of sandbox providers now sells exactly this, priced per VM-second like the cloud-desktop products of an earlier era. Composing the verified pieces, image tokens per step, seconds of inference, VM time, step counts inflated two to four times over the optimal path (Abhyankar et al. 2025), puts a GUI task's cost two to three orders of magnitude above the equivalent API calls, as an engineering estimate rather than a measured constant. That gap organizes the market: browser-only agents, driving a real browser through its debugging protocol where the DOM is available and instances pack densely, became the cheap middle tier, with full-desktop control reserved as the expensive general case. The operational details of sandboxes and their providers belong to Chapter 85; what belongs here is the design consequence, that computer use is bought by the trajectory, and every improvement in grounding pays for itself twice, once in success rate and once in steps not retried.

The screen is an injection surface

Prompt injection elsewhere in this book arrives through a retrieved document or a tool result. Here it arrives through everything, because the screen is the observation: page text, images, popups, hidden form fields, whatever an attacker can cause to be rendered. The vendor documentation says it without decoration: instructions found in content, on webpages or in images, can override the operator's instructions or cause mistakes (Anthropic 2024). The demonstrations are not hypothetical. Security researchers watched an agentic browser buy from a counterfeit storefront with stored payment details and follow instructions disguised as a CAPTCHA (The Hacker News 2025); the deployed mitigations, injection classifiers watching the screenshots, mandatory confirmation before consequential actions, human-typed credentials the model never sees, domain restrictions, are the runtime-safety machinery of Chapter 57 specialized to a channel that cannot be pre-filtered, and the payment protocols of Chapter 80 assume this layer leaks. A published red-team of one browser agent made the state of the art concrete: safeguards cut attack success from roughly a quarter of attempts to a tenth, and browser-specific attacks to zero, numbers that justify deployment gates rather than confidence (Anthropic 2025).

Further reading