Computer Use: the GUI as an Action Space

Computer use lets an agent reach software that lacks a suitable machine interface for the work at hand. It can operate a vendor portal, a desktop application, or a canvas whose useful controls are available only through the interface built for people. That reach is valuable, but indirect. A screen is an observation; pointer and keyboard events are actions. Neither a visible button nor a logged-in browser tells the harness whether the user authorized the effect behind the button.

The practical rule is therefore narrow: prefer the typed tool when an API, command, or semantic UI operation expresses the intended effect and its authorization. Use computer vision and coordinates where structure is absent, incomplete, or unreliable. Computer use is a universal fallback for rendered software, not a universal replacement for machine interfaces. A capable system can switch representations within one task, but every channel must feed the same run and effect ledger described in Chapter 41.

This chapter develops that fallback as a partially observed control loop. It separates seeing, grounding, acting, and verifying; defines the contract between a model and an executor; shows why retries can duplicate real effects; and explains how to evaluate the whole system without confusing a grounding score with successful work.

From scripted interfaces to learned control

Screen-driven automation predates current multimodal models. World of Bits exposed web pages through both pixels and the DOM, with mouse and keyboard actions, in 2017 (Shi et al. 2017). Traditional browser automation and robotic process automation use selectors, accessibility properties, or recorded coordinates. Learned computer-use systems add a policy that can interpret unfamiliar layouts and propose actions from visual or structured observations. Anthropic's 2024 public beta brought that pattern to a frontier-model product: screenshots entered the model, while a developer-supplied tool executed the requested mouse and keyboard actions (Anthropic 2024).

Learned grounding does not make interface drift disappear. A visual policy may survive a changed DOM path and fail after a theme, scale, icon, or layout change. A semantic locator may survive a cosmetic redesign and fail when a custom widget omits its accessible name. These methods fail under different changes. The useful question is not whether pixels or selectors are inherently robust, but which observation and action channel exposes enough stable identity for this operation.

A partially observed control loop

Let $s_t$ be the hidden application state at step $t$. It includes more than the visible page: active account, selected records, open dialogs, unsaved edits, remote service state, focus, clipboard, downloads, and windows outside the current viewport. The executor returns a partial observation

$$o_t = O(s_t, v_t),$$

where $O$ is the observation process and $v_t$ is capture metadata such as the window, viewport, resolution, scale, cursor, timestamp, and selected modalities. The observation may contain a screenshot, a DOM or accessibility-tree subset, detected elements, the current URL, and focus metadata. It is evidence about $s_t$, not the state itself.

Given the history $h_t=(o_0,a_0,r_0,\ldots,o_t)$, user instruction $u$, and remaining budget $b_t$, a policy proposes an action according to

$$\pi(a_t \mid h_t, u, b_t).$$

Here $\pi$ is the policy, and the expression denotes its distribution over the next action $a_t$ under that history, instruction, and budget. The executor validates and dispatches the selected action. The environment then changes according to

$$s_{t+1} \sim T(s_t, a_t, \xi_t),$$

where $T$ is the transition process and $\xi_t$ collects effects the agent does not control: network delay, animation, a popup, another actor, or a remote failure. A task succeeds only when an independent verifier establishes the required postcondition $G(s_T)$ before a step, time, or cost horizon. Every symbol in these equations names a component that an implementation must record or bound; none assumes that the model sees the hidden application state.

This formalism matters because a partial observation can be plausible and wrong. A screenshot can omit a dialog behind the current window. A tree can be stale. A click can succeed remotely just before the browser crashes. The model's statement that the task is done is another proposal, not proof.

One interaction step should therefore be explicit:

1. Capture an observation with an identity, timestamp, coordinate frame, window, focus, and selected modalities.
2. Let the model propose one action or a small, bounded batch.
3. Resolve the semantic intent, target origin, account, resource, payload, and expected effect.
4. Check authorization, approval, freshness, focus, coordinate bounds, and actionability outside the model.
5. Dispatch with an action ID and deadline.
6. Wait for an observable condition, not an arbitrary sleep, then recapture after each state-changing action.
7. Return executor evidence and verify the intended postcondition. Distinguish executed, rejected, timed_out, and outcome_unknown.

The boundary can be represented by three deliberately boring records:

UIObservation {
  frame_id,
  captured_at,
  window_id,
  viewport,
  focus,
  screenshot_ref,
  semantic_elements,
  visible_dialogs
}

UIAction {
  action_id,
  frame_id,
  window_id,
  viewport,
  kind,
  target,
  payload_ref,
  precondition,
  postcondition,
  risk_class,
  approval_id,
  timeout
}

UIResult {
  action_id,
  status,
  dispatched_at,
  completed_at,
  new_frame_id,
  observed_effect,
  remote_receipt,
  error
}

The executor rejects a stale frame, an invalid or out-of-bounds point, a focus mismatch, an expired approval, or a target whose identity has changed. If an image was downscaled before inference, coordinate remapping must use the recorded transform in both directions; current vendor guidance explicitly warns that coordinates refer to the original screenshot resolution (OpenAI 2026). Before a consequential click, recapture and revalidate the active window, origin, target, and exact payload. Batching stops at every side-effect boundary.

Pixels, structure, marks, and tools

A GUI system has more than two representations. Each reveals a different part of the state and supplies a different kind of target identity.

Screenshot pixels. They preserve visual layout, custom drawing, icons, spatial relationships, and canvas content. They struggle with small or occluded targets, resizing errors, the absence of stable element identity, and invisible or offscreen state.

DOM or accessibility tree. These expose names, roles, values, hierarchy, and sometimes node-addressed actions. They can omit custom widgets, include hidden or offscreen attacker text, return stale nodes or noisy dumps, and leave visual position ambiguous.

Detected elements or set of marks. These create a compact bridge from regions to stable labels for one frame (Yang et al. 2023). They inherit detector omissions, duplicate labels, overlays that obscure content, and movement that invalidates the labels.

API or typed tool. This channel supplies explicit arguments, validation, return values, and often idempotency or audit support. Its limits are coverage, integration work, schema drift, and operations the interface does not expose.

Coverage and identity are separate concerns. Pixels have broad visual coverage but weak identity. A tree can name an element while failing to show that it is covered by a modal. A typed operation can identify the resource precisely but omit a visual judgment the task requires. Structured observations are also untrusted content: a stale accessibility tree or attacker-controlled label is not safer merely because it is machine-readable.

Use the strongest channel that faithfully expresses the operation. Read a chart visually if its information exists only in the rendering. Use a semantic element action when its role and identity are current. Call an API for a payment or deletion when that API provides precise resources, authorization, idempotency, and receipts. A hybrid policy should be tested against pixel-only and structure-only alternatives under the same model, tasks, and harness; OSWorld's original ablations did not establish one representation as universally best (Xie et al. 2024).

Grounding, execution, and verification

"Click the Save button" hides at least five separate problems:

1. Intent and planning: decide that saving is the next useful operation.
2. Target identification: distinguish the document's Save action from similar controls.
3. Grounding: map that target to a point, node, or semantic handle in the current observation.
4. Execution: deliver the event to the intended window and control.
5. Outcome verification: establish that the correct document was saved, not merely that the button changed appearance.

ScreenSpot-Pro isolates the third problem. It evaluates point-in-target grounding in high-resolution professional applications across operating systems. Its strongest previously tested model reached 18.9%, while cascaded visual search reached 48.1% (Li et al. 2025). That result demonstrates substantial grounding difficulty in this setting. It does not prove that grounding dominates planning, operational knowledge, recovery, or verification in end-to-end work.

Small targets expose two independent engineering failures. Downsampling can erase useful detail; resizing without a consistent transform can shift every coordinate. Zooming, cropping, and higher-resolution tiles can help the first problem, while explicit coordinate frames solve the second. Current computer-use schemas include zoom actions for this reason, but zoom changes the observation and invalidates coordinates from the earlier frame (Anthropic 2026).

Reliability is reaching a verified state

For a fixed path of $N$ required steps, let $E_t$ mean that step $t$ achieved its postcondition. Define the conditional step probability

$$q_t = \Pr(E_t \mid E_{<t},h_t),$$

where $E_{<t}$ denotes all required step outcomes before $t$. The chain rule then gives

$$\Pr(\text{path succeeds}) = \prod_{t=1}^{N}q_t.$$

The familiar $p^N$ is only the homogeneous independent special case in which every conditional probability equals $p$ and any failure is unrecoverable. GUI errors are correlated: one wrong window can poison several later actions; a slow page can cause repeated stale clicks; a recovery action changes both the horizon and future probabilities. For an interactive policy, the useful quantity is the probability of reaching a verified goal state before its horizon, including recovery behavior.

OSWorld-Human makes efficiency visible alongside completion. In its evaluated systems, model calls dominated latency, later steps could take much longer than early ones, and the best agents used 2.7--4.3 times as many steps as the human-authored reference trajectories required (Abhyankar et al. 2026). These are measurements of particular agents and tasks, not a universal multiplier. They do show why task success alone hides operational quality.

Retries require effect semantics. A read, zoom, or fresh screenshot is usually safe to retry. A send, purchase, delete, permission change, or form submission may not be. If a timeout occurs after the remote effect but before the next observation, blind replay can duplicate it. A mutation is safe to retry only when it is idempotent or authoritative current state proves that the effect did not occur. Otherwise inspect and reconcile using a receipt, application history, or remote state. If the outcome remains ambiguous, enter needs_reconciliation rather than clicking again.

Approvals follow the same rule. An approval for an irreversible action is bound to the exact payload, target, origin, account, price or data disclosure, and current state. It is single-use and expires. A changed screen or payload requires a new approval; approval of a broad goal is not approval of every click used to pursue it. Stopping or restoring the local environment cannot undo an effect already committed to a remote service.

The environment is part of the system

Not every task needs a full desktop. A browser-only environment is enough for a web workflow; a full desktop is required when the task crosses native applications, system dialogs, or local files. Whichever boundary is chosen, the environment owns state that the model does not: browser profiles and cookies, downloads, clipboard, files, notifications, multiple windows, popups, extensions, locale, scale, and authentication challenges.

Keep three kinds of state distinct:

• Application state is the local and remote state changed by the user's task.
• Session state is the browser or desktop profile, credentials, files, clipboard, network identity, and environment configuration.
• Model context is the lossy history supplied to the policy.

Restoring model context does not restore application state. Restoring a VM snapshot does not reverse remote application state. Reusing session state can leak one run's data or authority into another.

Run computer use in a fresh, isolated browser context or desktop environment with the minimum necessary account and network access (OpenAI 2026; Anthropic 2026). Never attach the agent to a host browser profile containing personal history, extensions, saved cards, or a password manager. Disable shared host mounts and clipboard bridges unless the task specifically needs them. Keep browser-control, remote-debugging, and display endpoints private, authenticated, and scoped to one session. Downloads go to quarantine and are not opened or executed automatically; uploads and sharing pass through the same data-loss and approval checks as any other transmission.

Isolation also has layers. A fresh browser profile separates session data. A browser sandbox and site isolation constrain hostile renderers. A hardened container or microVM protects the host. Network and account policy constrain remote effects. None substitutes for the others, and an ordinary container is not a separate operating-system kernel (Souppaya et al. 2017; Chromium Project 2026).

The interface is an injection and privacy surface

Pixels, OCR, page text, DOM nodes, accessibility labels, email, PDFs, downloads, and tool results are untrusted data, not authority. They may describe what the agent should do, but only direct user instructions and external policy can authorize a new target, data flow, or side effect (OpenAI 2026). An authenticated browser carries ambient authority through its cookies. If page content can make the agent read one origin and type its data into another, the agent becomes a confused deputy across boundaries that normal browser code could not cross.

Authorization must therefore happen outside the model for every proposed effect. Resolve the authenticated user and tenant, current origin and account, target resource, semantic action, exact arguments, data being disclosed, remaining budget, and required approval. An allowed domain does not make its content trusted: it can host user text, third-party frames, redirects, malicious ads, or compromised pages. Navigation policy must recheck the scheme, host, port, resolved destination, redirects, and access to local, private, and metadata networks.

A prompt-injection detector can reduce exposure, but a classifier is not authorization. If it misses an attack, the policy layer must still prevent unauthorized sending, publishing, purchasing, deletion, installation, permission changes, acceptance of legal terms, and disclosure of sensitive data. WASP evaluates this end to end and finds that realistic, low-effort injections can divert capable web agents (Evtimov et al. 2025). Treat attack success, partial attack progress, and privileged effects as separate outcomes; failure to finish an attack may be security by incompetence, not a durable control.

Privacy starts before model inference. A screenshot or structured tree can contain names, account identifiers, notifications, input values, browsing history, one-time codes, and data outside the intended task. Capture the smallest relevant window or viewport, mask sensitive regions before upload where possible, and avoid raw screenshot logging by default. Store necessary artifacts encrypted, per session, with narrow access and short retention. Pause both model actions and capture during trusted user takeover.

Typing sensitive data counts as transmission (OpenAI 2026). Human typing alone does not guarantee secrecy: the next screenshot, DOM value, autofill popup, or account page can reveal it. Prefer a scoped credential or action broker. If a person must enter a password or one-time code, transfer control through a trusted surface outside the agent, suspend capture and model actions, clear transient UI, and resume with reduced authority. The model should not receive reusable credentials.

Place approval at the point of risk, immediately before the commit action. Show the person the exact site and account, recipient or target, payload, disclosed data, amount, and irreversible effect. Bind approval to an immutable action ID or payload hash, revalidate the fresh screen and origin after approval, and require another approval if anything changed. Text on the screen can never approve itself.

What a trajectory costs

Computer use consumes a trajectory, not one model call. A useful accounting identity is

$$C = \sum_{t=1}^{N} C_t,$$

where the cost of step $t$ is

$$\begin{aligned} C_t ={}& C_{\text{model},t}+C_{\text{image},t} \\ &+C_{\text{env},t}+C_{\text{tool},t}. \end{aligned}$$

with critical-path latency

$$L = \sum_{t=1}^{N} L_{\text{critical},t}.$$

Here $N$ is the number of attempted steps, including recovery; $C_{\text{model},t}$ is inference cost; $C_{\text{image},t}$ is image processing or image-token cost; $C_{\text{env},t}$ is browser, container, or VM-seconds; and $C_{\text{tool},t}$ covers capture, dispatch, storage, and verification. $L_{\text{critical},t}$ is the non-overlapped wall time on step $t$ across capture, transport, inference, dispatch, settling, and verification. Each term should be measured, not inferred from a universal GUI-to-API multiplier.

Batching can reduce model calls, but it spends freshness: the later actions are chosen without seeing intermediate state. Parallel actions are safe only when their resources and effects are independent. Report task success together with action count, model calls, tokens, wall time, tail latency, VM-seconds, retries, and human intervention. Report both all-attempt cost and cost conditioned on success; the latter alone hides expensive failures.

Evaluate the system you actually run

A grounding benchmark asks whether a model identifies the right target in a supplied image. An end-to-end task benchmark asks whether the model, harness, executor, environment, and verifier reach the required state. ScreenSpot-Pro measures the first. OSWorld measures the second on desktop applications: its original 369 tasks reported 72.36% for humans and 12.24% for the best tested agent (Xie et al. 2024). AndroidWorld adds 116 parameterized tasks across 20 mobile applications, with programmatic setup, success checks, and teardown; its original results also showed that task variations can materially change performance (Rawles et al. 2025).

These numbers are not points on one timeless curve. Benchmark tasks, environment images, application versions, observation and action interfaces, maximum steps, prompts, retries, and verifiers change. Original OSWorld and OSWorld-Verified results are not directly comparable, nor does a later agent score on a revised benchmark cross the original human baseline. OSWorld 2.0 deliberately changes the problem again: its 108 workflows are much longer and expose failures involving constraints, changing information, hidden state, asking the user, and verification (Yuan et al. 2026). The benchmark version and harness are part of every score.

Use a state-based verifier that reads authoritative application or service state. A final screenshot and the model's self-report are weak evidence. For each run, record the task and initial state, benchmark commit, environment image, application versions, resolution, scale, locale, account, network conditions, observation sources, action interface, model snapshot, prompt, harness and verifier versions, limits, retries, and exclusions. Report completion with a bootstrap confidence interval and stratify by application, task horizon, and risk where sample size permits.

A release evaluation should cover four families of evidence:

• Capability: task completion, grounding accuracy, invalid-action rate, stale or out-of-bounds actions, unnecessary steps, and model calls.
• Efficiency: wall time, tail latency, tokens, cost, VM-seconds, retries, and human intervention.
• Recovery: success after delayed rendering, moved or occluded targets, popups, focus theft, stale accessibility tree, network retries, authentication challenges, crashes, and an ambiguous duplicate action.
• Safety: side-effect precision, approval-bypass rate, prompt-injection attack success, sensitive-data exposure, post-cancel action count, unauthorized-origin attempts, and effects committed after timeout.

Test those faults deliberately. Vary resolution, scale, theme, locale, and window size. Insert animation, delayed loading, overlays, duplicate controls, focus changes, and stale structure. Crash after a click but before the result observation. Inject an instruction into visible content and into a structured label. A system that passes only the clean path has not demonstrated reliable computer use.

A deployment checklist

Before exposing a GUI action surface, verify that:

• the typed or semantic alternative was considered first;
• every observation has a frame, window, viewport, scale, focus, and timestamp;
• coordinates and semantic handles are checked against fresh state;
• every mutation declares a precondition, postcondition, risk class, and retry policy;
• approvals bind to the exact target, payload, account, and current state;
• ambiguous outcomes enter reconciliation instead of blind replay;
• the browser or desktop is isolated from host profiles, credentials, files, and control endpoints;
• capture, logs, clipboard, downloads, uploads, and retention have explicit privacy rules;
• an independent verifier checks current state; and
• evaluation reports the complete model--harness--environment configuration.

One agent already faces partial observation, stale state, and ambiguous effects. Adding agents does not remove those conditions. If several policies share a browser or desktop, they can race on focus, invalidate one another's frames, and duplicate side effects unless ownership and coordination are explicit. Chapter 43 takes up that coordination problem next.

Further reading