Computer Use and Browser Agents

click(x=160, y=118)

A weather API, calculator, or retrieval endpoint can expose named operations with argument schemas. The model requests one of those operations, and the runtime validates and executes it.

Some applications have no suitable automation API: a legacy expense tool, an internal portal, or a specialized desktop program. Their visible interface may still be available. Computer use lets an agent request pointer and keyboard actions against that interface.

Working through the GUI

A function tool works through a defined application interface. A GUI tool instead works through visible controls: observe the screen, move the pointer, type, and click.

To extract a table from an interactive dashboard, for example, the agent may need to open a menu, set a filter, and read the updated page. Screenshot-based computer use requires locating those controls and checking the result of each action.

A screenshot is the model’s current visual observation, not the entire state of the computer. The user’s goal, retained history, and other tool results also matter. A hidden dialog or an offscreen control may require another observation before the model can choose an action.

What the loop actually looks like

The outer shape is the agent loop from earlier in this arc. The tools expose a small vocabulary of screen capture, pointer, keyboard, scrolling, and waiting operations. The exact packaging varies: one tool may dispatch several action types, or an API may expose them as separate tools. An action result may include a screenshot, an acknowledgement, or an error.

click(x=265.6, y=34.0)

Coordinate actions depend on the screenshot’s coordinate system. After an action, the runtime needs some way to determine whether the UI is ready for another observation, such as an explicit wait, a load signal, or a fresh screenshot.

Waiting and observing are part of the control loop: an action issued against a stale screen can target the wrong element.

Prerequisites and scope

This post uses the agent loop and function calling from earlier in the arc, and introduces a multimodal observation:

• The agent loop. Generate, parse, execute, resume. See the agent loop for how calls and observations move between turns.
• Function calling as structured generation. The model is still emitting tool calls. The tool just happens to be a computer. See function calling.
• Images as observations. A vision-capable model receives a processed representation of the screenshot alongside text. The image encoding and resolution policy depend on the model; this post focuses on the resulting action loop.

We're scoped to general-purpose computer use: Anthropic's computer tool, OpenAI's CUA / Operator, browser-agent libraries like Browser Use. Specialized SWE-agents share the same loop shape but operate on terminals and editors rather than GUIs; this post concentrates on the screenshot-and-GUI interface.

The formalism, kept minimal

You can think of computer use as a partially-observable Markov decision process where the state is the computer's true state, the observation is a screenshot, and the action space is a small set of action types with parameters such as coordinates and text. The policy conditions each action on the observations available so far.

The action space is roughly:

The observation at step $t$ is a screenshot $o_t$, a 2D grid of pixels (in practice an image whose format and resolution depend on the harness). The agent's policy is the model itself:

That is, conditioned on the user goal $g$, current screenshot, and retained action/observation history, the model samples the next action.

The hard part is that $o_t$ is pixels, not a clean feature vector, and $\pi_\theta$ has to localize a button inside it well enough to emit a coordinate that lands on it.

Pointer and keyboard actions

The tool definition uses a small set of pointer and keyboard operations. key handles copy, paste, tab, escape, arrow keys, Cmd-T for a new tab. click plus scroll cover the vast majority of pointer interaction. type just types literal text wherever focus currently is.

The same operations work across many interfaces. A web form with 30 fields is still just a sequence of clicks and types. A complex menu is a click that opens a submenu, then another click. Once you commit to "the model acts at the cursor-and-keyboard level," the action space is fundamentally small. What scales is the composition of actions into a plan, not the vocabulary.

The cost of that small vocabulary is that every semantically meaningful operation becomes a sequence. Selecting text may require several pointer events or one drag action, depending on the tool. Filling a form is click on the field, type the value, key(Tab), repeat. Each additional model turn adds token and latency costs.

The cost of observing between actions

A simple harness takes a screenshot, asks for one action, executes it, and repeats. That serial design pays model latency at every step. It is not a universal API requirement: some computer-use interfaces return action batches, and browser harnesses can group deterministic operations.

Measure model latency, screenshot processing, execution, and UI waits separately before choosing what to optimize.

Waiting for model output between small UI operations can make even a short task slow. Batching safe, predictable actions reduces round trips, but acting too far ahead without observations increases the risk of using a stale UI state.

Take another screenshot when the next action depends on what the last one changed; use a deterministic sequence when its intermediate state is already known and checked.

Screenshots, the context window for the world

In a screenshot-driven harness, the model’s information comes from the user’s goal, supplied screenshots, and whatever actions, tool outputs, or summaries the runtime retains.

Image processing can remove information. A two-pixel focus ring may become indistinguishable after downscaling. A tooltip that faded out 200ms before the capture is gone. The status of a checkbox that's offscreen requires a scroll plus another call. Another screenshot, a scroll, or a change in resolution may be needed to observe the missing detail.

The transcript does most of the remembering. The runtime chooses which screenshots, actions, tool outputs, and summaries to retain. Image-token cost depends on the provider’s image processing and resolution; there is no single token count for a 1280 × 800 screenshot.

The model has to localize coordinate targets. “Click Submit” needs a position in the displayed image. Changing resolution, cropping, or the device scale can invalidate coordinates unless the runtime transforms them consistently.

Under the simulation’s fixed-noise assumption, a smaller target catches fewer clicks. Real grounding errors also depend on layout, resolution, target identity, and the model; the example isolates only target size.

Pixels vs the DOM

For a browser, the harness can supply structured DOM or accessibility information and assign references to actionable elements. Then the model can request click(id=5) without estimating pixel coordinates. Browser-agent libraries can combine this with screenshots.

Structured browser observations can reduce the need for visual localization. Screenshots remain useful for canvas content, visual layout, and interfaces without meaningful accessibility nodes. A hybrid harness can choose between these surfaces. Set-of-Mark prompting supplies numbered visual targets as another option.

A minimal implementation

A computer-use harness must preserve the assistant’s tool calls and return one result for each call ID. Here is the client loop for an Anthropic Messages-style adapter. generate handles the selected model and computer-tool version; execute is the VM or browser driver. These are application-supplied functions, so the snippet is a harness sketch rather than a complete desktop implementation.

import base64
 
def screenshot_content(png_bytes):
    return [{
        "type": "image",
        "source": {
            "type": "base64", "media_type": "image/png",
            "data": base64.b64encode(png_bytes).decode(),
        },
    }]
 
def run_computer_task(prompt, initial_png, generate, execute, max_turns=20):
    messages = [{"role": "user", "content": [
        {"type": "text", "text": prompt},
        *screenshot_content(initial_png),
    ]}]
    for _ in range(max_turns):
        response = generate(messages)
        messages.append({"role": "assistant", "content": response.content})
        if response.stop_reason == "end_turn":
            return "".join(b.text for b in response.content if b.type == "text")
        if response.stop_reason != "tool_use":
            raise RuntimeError(f"Incomplete turn: {response.stop_reason}")
        results = []
        for block in response.content:
            if block.type != "tool_use":
                continue
            # Validate actions and permissions in the driver. Run in order.
            # Return content blocks: screenshot, acknowledgement, or error.
            content, is_error = execute(block.name, block.input)
            results.append({
                "type": "tool_result", "tool_use_id": block.id,
                "content": content, "is_error": is_error,
            })
        messages.append({"role": "user", "content": results})
    raise RuntimeError("Model-turn budget exhausted")

The initial screenshot can be a user image. Subsequent observations produced by a requested tool belong in its matching tool_result. Append the assistant’s content, not the whole API response object: response metadata is not a transcript message.

Where it keeps falling apart

A short successful demonstration does not establish reliability on a longer workflow. Task-level evaluation needs to include failed actions, recovery, and verification of the final state.

If each necessary step succeeds independently with probability 0.95 and any failure ends the run, twenty steps all succeed with probability 0.95²⁰ ≈ 36 percent. Real tasks violate those assumptions: failures can be correlated, actions vary in difficulty, and agents can recover.

One way to limit that risk is to divide work into bounded tasks with explicit checks between them. A checker can verify a saved file, a submitted form’s confirmation, or a changed setting. Whether this improves success must be evaluated alongside the cost of recovery.

Safety: blast radius and the sandboxing pattern

A computer-use agent can request actions within the account permissions, exposed tools, and restrictions of its environment. A broadly privileged desktop gives it a much larger action surface than a small set of application-specific functions.

Provider guidance recommends isolating the environment, limiting exposed applications and credentials, and asking for confirmation on consequential actions. The harness must implement those boundaries and decide what to record for auditing. A disposable VM can limit damage to that VM; it does not by itself restrict the remote accounts available inside it.

Misconceptions

"Computer use is OCR plus clicking." Reading text is only one part. The agent also needs to identify focus, loading state, modal dialogs, and whether an action had the intended effect. A readable button label does not establish that clicking it is the right next action.

"A screenshot is a complete representation of the state." A form may reveal a validation error only after submission, and offscreen controls require another observation. The agent must distinguish what it has observed from what it assumes about the interface.

"DOM-based agents are obsolete." Structured observations remain useful when a page exposes meaningful elements and accessibility information. Screenshots can supply visual evidence that those structures omit. The appropriate mix depends on the application and harness.

"A benchmark score establishes reliability for my workflow." Benchmark results depend on the task distribution, environment, step budget, model, and harness. Test the complete workflow you intend to automate, including recovery and verification, rather than borrowing a success rate from a different setup.

What's next

The next post covers multi-agent systems: how to coordinate workers with separate contexts and scoped responsibilities. Delegation changes where work and context accumulate, but adds its own coordination costs.

Additional reading (and watching)

• Yang, J., et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. Introduces the "agent-computer interface" framing and argues that the shape of the action space is itself a design lever, not a fixed given.

• Anthropic. (2024-2026). Computer use (beta) — tool definitions and loop guide. The canonical reference for the computer_20250124 tool, its action vocabulary, and the recommended outer loop structure.

• Anthropic. (2024). Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. The original release announcement.

• OpenAI. (2025). Introducing Operator and the Computer-Using Agent (CUA). Product post and CUA model card. Describes the browser-only sandbox, the Set-of-Mark-influenced prompting, and the human-confirmation pattern for irreversible actions.

• Browser Use. (2024-2026). Browser Use — AI-first browser automation library. Open-source library implementing the DOM-first, pixel-fallback hybrid for browser agents. A reference for combining browser observations and actions.

• Yang, J., et al. (2023). Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv:2310.11441. The paper that showed overlaying numbered labels on a screenshot makes multimodal models much better at emitting actions keyed to those labels. A practical middle ground between pixels and DOM.

• Xie, T., et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024. Full-desktop computer-use benchmark, not just browser. A benchmark for end-to-end desktop tasks under a specified environment and evaluation protocol.

• Zhou, S., et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854. Self-hosted web environments designed to measure agent success on realistic multi-step browser tasks.

• Mialon, G., et al. (2023). GAIA: A Benchmark for General AI Assistants. arXiv:2311.12983. It evaluates assistant tasks requiring combinations of reasoning, information gathering, and tools; it is not exclusively a GUI benchmark.

• Zheng, B., et al. (2024). GPT-4V(ision) is a Generalist Web Agent, if Grounded. ICML 2024. Early study of using vision-language models as web agents. Examines how grounding actions in web elements affects task performance.