Computer Use and Browser Agents

Most of the agent stack we've built up in this arc assumes the tools are built for the agent. A weather API, a code interpreter, a calculator, a retrieval endpoint. Each one is a clean function with a JSON schema, and the model's job is to call it. That world is tidy because the tool designer and the agent builder can agree on a contract in advance.

Now look at your laptop. Most of the software on it has no agent-friendly API. A legacy expense tool, the internal HR portal, a niche CAD program, a video editor, the mail client your company insists on running in a web view. These systems already have an interface. It's the screen. The buttons, the menus, the forms, the cursor. That interface is universal in one sense, since every human user goes through it, and hostile in another, because it wasn't designed to be consumed by a language model.

Computer use is what happens when you point the model at that interface anyway.

The only universal interface is the GUI

The tidy version of tool-using agents stops at the edge of whatever systems decided to publish an MCP server or a function-calling endpoint. Everything else, which is to say most software, has to be driven the same way a human drives it. You see the screen, you move the cursor, you type, you click.

If you want an agent to book a flight on an airline whose site predates the agent era, or file an expense in a SaaS app that hasn't shipped an API in a decade, or scrape a table out of an interactive dashboard, the only surface that reliably exists is the pixel grid. So the question the field has been poking at since late 2024 is: can a language model drive a GUI the way a person does, through nothing but screenshots and cursor actions?

The sticky handle I carry around for this: the screen is the model's context window for the world. Every turn, the model gets a JPEG of whatever's on the display. That image is the totality of its observation. Everything else (modal dialogs it missed, scroll position it can't sense, a tooltip that disappeared half a second before the frame) is outside the context window as surely as a token past the 128k cutoff.

What the loop actually looks like

Before we get into the mechanics, a picture. The outer shape is exactly the agent loop from earlier in this arc, but the tool list is short and weird. One tool, really: computer. Its inputs are screenshot and a small action vocabulary (click, type, key, scroll, wait). Its output, always, is the new screenshot after the action has been executed.

A few things to notice in that trace. First, the "thought" on each step isn't free. Those are the model's own chain-of-reasoning tokens, emitted before the action and counted against the same budget. Second, the action is a coordinate, not a CSS selector. The model isn't saying "click the Search button"; it's saying "click pixel (920, 64) on this specific frame". Third, after every action there's a forced wait for the page to settle, because the model has no other way of knowing whether its click actually did anything.

That third point is where most of the fragility lives. I'll come back to it.

Prerequisites and scope

This post sits on top of three things earlier in the arc:

• The agent loop. Generate, parse, execute, resume. If that ring doesn't already feel native, start at the agent loop. Computer use is a specialization of it, not a new idea.
• Function calling as structured generation. The model is still emitting tool calls. The tool just happens to be a computer. See function calling.
• Multimodal vision basics. The model is reading images, not the DOM. You should be comfortable with the mental model of "the screenshot is tokenized into image patches that live alongside the text tokens in the context window."

We're scoped to general-purpose computer use: Anthropic's computer tool, OpenAI's CUA / Operator, browser-agent libraries like Browser Use. Specialized SWE-agents share the same loop shape but operate on terminals and editors rather than GUIs; that's a relative of computer use, not computer use itself.

The formalism, kept minimal

You can think of computer use as a partially-observable Markov decision process where the state is the computer's true state, the observation is a screenshot, and the action space is a small discrete set. I'll write it down once because it's useful to see the shape, and then we'll never use the notation again.

The action space is roughly:

The observation at step $t$ is a screenshot $o_t$, a 2D grid of pixels (in practice a downscaled JPEG, typically around 1280 × 800). The agent's policy is the model itself:

which is a fancy way of saying: conditioned on the current screenshot plus everything that came before in the transcript, the model samples the next action. The transcript is the memory. The screenshot is the world.

The hard part is that $o_t$ is pixels, not a clean feature vector, and $\pi_\theta$ has to localize a button inside it well enough to emit a coordinate that lands on it.

The action space is weirdly small

When I first read through the Anthropic tool definition, I was surprised by how short the action vocabulary is. A handful of verbs covers almost everything a human does at a keyboard and mouse. key handles copy, paste, tab, escape, arrow keys, Cmd-T for a new tab. click plus scroll cover the vast majority of pointer interaction. type just types literal text wherever focus currently is.

Why such a small vocabulary? Because the GUI itself is already the compression layer. A web form with 30 fields is still just a sequence of clicks and types. A complex menu is a click that opens a submenu, then another click. Once you commit to "the model acts at the cursor-and-keyboard level," the action space is fundamentally small. What scales is the composition of actions into a plan, not the vocabulary.

The cost of that small vocabulary is that every semantically meaningful operation becomes a sequence. Selecting text means click-down, drag, click-up, three actions. Filling a form is click on the field, type the value, key(Tab), repeat. The agent spends most of its tokens composing primitives, not choosing among rich verbs.

One action, one model call

Here's the thing that determines almost everything about the feel of computer use: every action is one model call. There is no batching, no lookahead, no "predict the next three actions." The loop forces a round-trip for each primitive.

The model-call slice dominates. Screenshot capture is cheap (a few hundred milliseconds at most). Action injection (the runtime telling the VM to click) is a rounding error. Page settle varies, but most UIs are fast enough once the click lands. The model call is the bottleneck, because it's a full inference pass over an image that's been tokenized into thousands of image tokens, plus the transcript of prior steps, with enough reasoning budget to localize the next target.

That's why computer use feels slow to watch even when it's working. You're watching a model think in real time, once per action. A person who knows the task can do eight clicks in ten seconds; the agent will take forty. Streaming responses help the feel but don't change the math. Every primitive costs one inference.

This is the "speed-of-light" cost I carry around for computer use. Until the loop itself changes (one model call emitting multiple actions, or action prediction handed off to a tighter, cheaper policy model rather than the full-fat reasoner), the wall-clock is gated by this ratio.

Screenshots, the context window for the world

The model's entire knowledge of the state of the computer at step $t$ is the screenshot and its own prior outputs.

The screenshot is lossy in ways the model can't fix. A focus ring that's two pixels wide might render as one pixel after downscaling, and the model genuinely cannot see it. A tooltip that faded out 200ms before the capture is gone. The status of a checkbox that's offscreen requires a scroll plus another call. This is a real source of error, and the usual recovery path is: take another screenshot, or try an action and see what happens.

The transcript does most of the remembering. The model doesn't keep a running belief state about the world. It keeps a conversation. The last 10-20 screenshots plus their associated thoughts and actions are in-context; everything older is either summarized or gone. Most providers clip the history of images aggressively because image tokens are expensive. A 1280 × 800 screenshot typically weighs in around 1,500–3,000 tokens after tokenization.

The model has to localize every step. When a human user says "click the Submit button," their visual system has already done the localization. The model has to do it explicitly, every time, because its output is a coordinate. That localization is where most of the frontier-model improvements have gone since the first Claude 3.5 Sonnet computer-use release. Anthropic's computer-use training explicitly included synthetic GUI localization data for exactly this reason. OpenAI's CUA took a similar route, fine-tuning a specialist variant on browser trajectories.

The thing to take away from that scatter is that the pointing noise doesn't scale with the button. The model has roughly a fixed-pixel-variance "aim," and as the target shrinks, it starts missing more often even though nothing else changed. Real GUIs are full of small affordances (window controls, toolbar icons, close buttons on tabs, spinner arrows in number inputs), and every one of them is a place the agent can fumble.

Pixels vs the DOM

When the target is specifically a web browser, you don't have to use pixels. The browser already exposes the structured representation of the page. You can feed the model a numbered list of accessibility nodes and let it emit click(id=5) instead of click(x=920, y=64). That's how Browser Use, Playwright-driven agents, and most of the "web agent" research papers work.

DOM-based agents aren't obsolete. DOM is faster, cheaper, and more reliable whenever the target page is a well-formed web app. Pixels are what you reach for when the DOM doesn't exist or lies: canvas-based editors, remote desktops streamed over VNC, apps rendered entirely in WebGL, iframes from domains that don't expose accessibility nodes. The production answer in 2026 is: try DOM first, fall back to pixels when DOM doesn't help. The Browser Use library, Stagehand, and the production browser harnesses inside the big labs all do some version of this hybrid. Set-of-Mark prompting, which overlays numbered labels on a screenshot so the vision model can pick a label rather than a coordinate, is a clever middle ground. It combines the robustness of pixels with the discreteness of DOM ids.

A minimal implementation

The Anthropic Messages API exposes computer use as a single server-defined tool. You don't write the tool schema yourself; you declare the tool type, the display dimensions, and your loop takes care of running the action on a VM you control.

import base64
import anthropic
 
client = anthropic.Anthropic()
 
def step(screenshot_png_bytes: bytes, transcript: list) -> dict:
    """One iteration: send the current screenshot, get the next action."""
    img_b64 = base64.standard_b64encode(screenshot_png_bytes).decode()
 
    # Append the fresh screenshot as a user message.
    transcript.append({
        "role": "user",
        "content": [{
            "type": "image",
            "source": {"type": "base64", "media_type": "image/png", "data": img_b64},
        }],
    })
 
    resp = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        tools=[{
            "type": "computer_20250124",   # server-defined tool
            "name": "computer",
            "display_width_px": 1280,
            "display_height_px": 800,
        }],
        messages=transcript,
        betas=["computer-use-2025-01-24"],
    )
 
    # The model either emits a tool_use block with an action, or plain text (done).
    for block in resp.content:
        if block.type == "tool_use" and block.name == "computer":
            return {"action": block.input, "stop": False, "transcript": transcript + [resp.to_dict()]}
    return {"action": None, "stop": True, "transcript": transcript + [resp.to_dict()]}
 
# The loop wrapper (elided): take screenshot → step() → execute action on the VM →
# take next screenshot → step() again. Exit when stop=True or a step budget is hit.

Two things to flag. First, computer_20250124 is a server-defined tool, which means the schema for click, type, and friends lives on Anthropic's side and you don't re-declare it. Second, the image goes in a user message, not a tool_result, because the screenshot is how you hand the agent fresh observations each turn rather than tool output. The loop around this is boring on purpose: screenshot, call, dispatch the returned action to your VM driver, loop.

Where it keeps falling apart

Every demo you'll see of computer use is a well-chosen, short task. The benchmarks tell a less flattering story about long-horizon reliability.

The shape of that chart is the shape of the problem. Single-step tasks are near-solved. Three-to-five-step tasks are usable with human oversight. Once you get to 10+ step workflows, success rates fall into the 20-40 percent range on OSWorld and WebArena, and progress has been slower than the demo trajectory suggested. The reason is simple: if each step has a 95 percent success rate (which is already optimistic for GUI localization), then 20 steps compound to 0.95²⁰ ≈ 36 percent. Recovery from a misclick is possible but not free, because the agent has to notice something went wrong first, and noticing from a screenshot is itself a task.

The practical answer in production systems hasn't been to push per-step accuracy to 99.5 percent. It's been to shorten the horizon: decompose the task, let the agent do bounded sub-tasks with explicit verification between them, and keep a human or a deterministic checker in the loop at every commit point. This is the GAIA benchmark intuition made operational.

Safety: blast radius and the sandboxing pattern

A computer-use agent can do anything a user can do on the machine. Install software. Send email. Move money. Delete files. Post to social media. Configure DNS. That makes the safety architecture fundamentally different from a normal tool-use agent, where the available tool set is what you explicitly enumerated.

The production pattern has converged on a few moves. Run the agent inside a disposable VM or container so the blast radius is bounded. Allow-list the sites and apps the agent is permitted to touch. For any action that's irreversible (sending a message, submitting a payment, modifying cloud resources) require an explicit confirmation from the human before executing. Record the full screenshot-and-action trajectory so you can audit after the fact. Anthropic's and OpenAI's own docs are explicit about all of this, and it's why the most polished demos still run inside a sandboxed browser tab rather than on your real desktop.

The reason these agents feel useful for "book me a flight" inside a hosted sandbox and raise serious safety concerns as a feature on your real laptop is the same reason: the interface is universal. When the tool is "do anything a human could do at this keyboard," the surface area of what can go wrong is the surface area of the OS. Shrinking that surface back to something auditable is what 2026 is currently spending its engineering budget on.

Misconceptions

"Computer use is OCR plus clicking." OCR is the easy half. The hard half is reasoning about UI state: which tab is active, whether a modal is blocking input, whether the page is still loading, whether the field you think is focused actually has focus. Most failure modes aren't "the model couldn't read the button." They're "the model read everything correctly and still emitted an action that doesn't make sense for this state," because UI state is contextual in ways a screenshot alone under-specifies.

"A screenshot is a complete representation of the state." It isn't, in a bunch of subtle ways. Focus rings can be invisible after downscaling. Scroll position below the fold is literally outside the observation. A disabled button looks barely different from an enabled one. A form with a client-side validation error may show nothing until submit. The agent has to develop something like a theory of the UI to interpret what the pixels imply, and that theory lives implicitly in the model's training.

"DOM-based agents are obsolete." Not even close. When the target is a normal web app, the DOM is faster, smaller, and more reliable than the screenshot. Most production browser stacks in 2026 use DOM first and fall back to pixels for canvas, WebGL, cross-origin iframes, and remote content. Vision-only is the universal fallback, not the default answer.

"We're close to general computer use." On the demo axis, it looks that way. On the reliability axis, we're at 20-40 percent end-to-end success on 10+ step tasks in real environments. The gap between a demo and a production system you'd trust with your inbox is still roughly an order of magnitude of reliability, and progress has been linear across model generations rather than the step-function the demos suggest.

What's next

The next post covers multi-agent systems: what happens when you stop trying to make one agent do everything end-to-end and instead coordinate several agents, each with a scoped role, a shorter horizon, and a narrower action space. That framing turns out to be an effective response to the same long-horizon reliability problem that keeps computer use under 40 percent on OSWorld today.

Additional reading (and watching)

• Yang, J., et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. Introduces the "agent-computer interface" framing and argues that the shape of the action space is itself a design lever, not a fixed given.

• Anthropic. (2024-2026). Computer use (beta) — tool definitions and loop guide. The canonical reference for the computer_20250124 tool, its action vocabulary, and the recommended outer loop structure.

• Anthropic. (2024). Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. The original release announcement. Worth reading alongside the current docs to see how the framing evolved.

• OpenAI. (2025). Introducing Operator and the Computer-Using Agent (CUA). Product post and CUA model card. Describes the browser-only sandbox, the Set-of-Mark-influenced prompting, and the human-confirmation pattern for irreversible actions.

• Browser Use. (2024-2026). Browser Use — AI-first browser automation library. Open-source library implementing the DOM-first, pixel-fallback hybrid for browser agents. Good reference implementation of the patterns most production stacks have converged on.

• Yang, J., et al. (2023). Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv:2310.11441. The paper that showed overlaying numbered labels on a screenshot makes multimodal models much better at emitting actions keyed to those labels. A practical middle ground between pixels and DOM.

• Xie, T., et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024. Full-desktop computer-use benchmark, not just browser. The realistic upper bound on what current agents can do end-to-end.

• Zhou, S., et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854. Self-hosted web environments designed to measure agent success on realistic multi-step browser tasks.

• Mialon, G., et al. (2023). GAIA: A Benchmark for General AI Assistants. arXiv:2311.12983. The benchmark most cited for "real tasks a human assistant should be able to do." Computer-use agents sit noticeably below human performance here, which is both humbling and useful.

• Zheng, B., et al. (2024). GPT-4V(ision) is a Generalist Web Agent, if Grounded. ICML 2024. Early study of using vision-language models as web agents. The "if grounded" caveat is load-bearing and foreshadows the DOM-vs-pixel debate this post spends time on.