The Agent Loop: Model, Runtime, Tool, Resume

An agent runtime repeatedly calls the model, executes requested tools, and returns their results as context. Each result gives the model information it can use on the next turn.

In the previous post we saw how function calling works as structured generation: the model emits a tool_use block, the provider returns structured arguments for the runtime to validate, and you get back a request to run a tool. A turn may request one or several calls. For a client-side tool, the application must execute the call and return its result in another request.

A longer task may need several turns: for example, reading a file, searching for missing information, and calculating a result before answering. The agent loop connects those completions and tool executions, carrying their results into subsequent calls.

A client-side tool needs another turn

With a client-side tool, a generation request can produce a final answer, request tool execution, or stop for another reason such as an output limit. Anthropic reports a stop_reason; OpenAI Responses represents calls as output items and also reports response status. Those are different API contracts around the same loop.

For client-side functions, your application runs the requested code and sends the result back. Provider-hosted tools are a separate case: an API request can include several internal tool steps before returning. A generation turn itself normally contains many token-by-token forward passes, not one forward pass.

Prerequisites

A short history of agents introduces tool-using language models; function calling as structured generation explains their structured requests. MCP later covers a shared host-to-tool interface. The caching material in Arc 5 explains how reusing prefix computation can reduce work as the transcript grows.

The minimal loop

A bounded loop with one tool-result message per call:

messages = [{role: "user", content: prompt}]
for iteration in range(MAX_ITERATIONS):
    response = model.generate(messages, tools=TOOLS)
    if response.stop_reason == "end_turn":
        return response.text
    if response.stop_reason != "tool_use":
        raise IncompleteTurn(response.stop_reason)
    messages.append({role: "assistant", content: response.content})
    results = []
    for block in response.tool_use_blocks:
        result = validate_and_run(block.name, block.arguments)
        results.append({type: "tool_result", tool_use_id: block.id, content: result})
    messages.append({role: "user", content: results})
raise IterationLimit()

A runtime can extend this loop with retries, timeouts, parallel execution, context compaction, and logging.

The four-state cycle

The loop has four states, with different parts of the application responsible for each:

1. GENERATE — the model emits tokens, possibly ending in a tool_use block.
2. PARSE — the runtime reads the stop reason and extracts tool calls.
3. EXECUTE — the tool (your code, a container, an API, whatever) runs and produces a result.
4. RESUME — the runtime appends the result to the transcript and calls the model again.

On each GENERATE step, the model receives the context assembled for that turn. It may include tool results, summaries, retrieved memory, and preserved reasoning items. Some APIs store prior state and accept a reference instead of requiring the client to resend it; either way, the model needs that state made available to the next generation.

What the runtime actually does

Between model generations, the runtime handles dispatch, validation, execution, and context assembly.

Dispatch. When the model emits a tool_use block with name: "search", something has to look up search in a registry of callables and invoke it with the provided arguments. Simple when you have two tools, less simple when you have an MCP server advertising sixty of them and you need to route by namespace.

Argument validation. Supported strict modes constrain arguments to a schema; ordinary tool declarations do not provide a portable guarantee. Validate application constraints and permissions before execution. A recoverable validation failure can become a tool error for the model; the runtime may instead stop when policy requires it.

Timeouts and retries. Tools may be local functions or remote calls, and either can fail or hang. The runtime wraps each EXECUTE in a timeout, and on safely retryable transient errors retries with backoff. Mutating tools need idempotency or outcome checks before retrying, because a timeout can happen after the side effect succeeded. For a recoverable exhausted retry, the runtime can append {error: "tool X failed after 3 retries"} as the tool result and let the model choose a permitted next step.

Error surfacing. Return useful, sanitized details for recoverable failures so the model can change arguments, choose another tool, or explain the failure. The runtime should stop on failures it cannot safely recover from.

Context assembly. The runtime decides which turns go into the next prompt. Every turn? A valid recent suffix? A summarized version of the first 20? This determines what evidence the model can use and we come back to it below.

Parallel execution. When the model emits several tool_use blocks in one turn (Anthropic's Messages API allows multiple; OpenAI's Responses API calls this "parallel tool calls"), the runtime can run them concurrently rather than one at a time. A question like "what's the weather in Tokyo, Paris, and New York?" is three independent calls.

How a loop ends

A bounded runtime needs a completion condition and a way to stop work that does not complete. The three examples below use a model stop reason, an iteration cap, and an error threshold; time or token budgets are other options.

end_turn. In this Anthropic example, the model reports a completed turn. The runtime returns the answer. Other response blocks may accompany the text; completion should be checked using the API contract rather than inferred from text alone.

max_iterations. A model can keep searching when the available sources do not contain an answer. An iteration cap bounds the number of turns spent on that unsuccessful trajectory.

error_threshold. A runtime can stop after a configured number of consecutive or recent failures. For example, repeated database connection failures may justify ending the run instead of issuing the same query again.

Context is the budget you actually spend

Tool results can dominate the context. For example, five assumed 40,000-token outputs already fill a 200,000-token window before the user prompt, tool definitions, or answer. The runtime must reduce, summarize, or retrieve material selectively when the assembled input would exceed the selected model’s limit.

Several context strategies can be combined.

• Summarize earlier turns once they're no longer actively informative. A checkpoint can preserve decisions, unresolved work, and evidence while replacing older bulk output.
• Drop tool results once the model has synthesized them into a claim. If turn 3 fetched a file and turn 4 said "the function is defined on line 42," you may be able to clear the bulk content. Preserve required tool-call/result pairing and any evidence needed to verify the claim.
• Externalize to memory. Write tool outputs to a scratchpad file or a vector store the agent can query on demand, instead of leaving them inline.

A three-step worked example

Suppose the user asks: "What is 7% of last year's revenue?" The agent loop plays out like this:

Iteration 1. GENERATE → the model, seeing the user's question and a tool schema that includes search and calculator, emits search("annual revenue 2025") as a tool_use. PARSE → runtime pulls the name and arguments out of the response. EXECUTE → runtime calls the search API, gets back "Revenue 2025: $412.0M". RESUME → runtime appends the assistant turn and the tool result to the messages list, calls the model again.

Iteration 2. GENERATE → the model now sees the revenue number. It knows 7% is a multiplication problem and it has a calculator, so it emits calculator(412_000_000 * 0.07). PARSE, EXECUTE, RESUME as before. The result 28_840_000 gets appended.

Iteration 3. GENERATE → the model has the multiplication result. No more tools needed. It emits plain text: "About $28.84M." stop_reason is end_turn. The loop exits. The runtime returns the final text.

The state-machine diagram above follows these two tool calls and the final generation.

Prefix-cache hits can reduce repeated prefill work, but require the provider’s caching rules to be satisfied and do not eliminate decoding or new tool-result processing. Measure each phase before choosing what to optimize.

Implementation sketch

Here is a small Anthropic Messages loop. Search is a fixed stub in this example, and production use needs additional timeout, authorization, and recovery behavior.

import anthropic, operator
 
client = anthropic.Anthropic()
 
TOOLS = [
    {
        "name": "search",
        "description": "Web search",
        "input_schema": {"type": "object", "properties": {"q": {"type": "string"}}, "required": ["q"]},
    },
    {
        "name": "calculator",
        "description": "Multiply two numbers",
        "input_schema": {
            "type": "object",
            "properties": {"a": {"type": "number"}, "b": {"type": "number"}},
            "required": ["a", "b"],
        },
    },
]
 
def run_tool(name, args):
    if name == "search":
        if set(args) != {"q"} or not isinstance(args["q"], str):
            raise ValueError("search requires a string q")
        return "Revenue 2025: $412.0M"  # fixed teaching stub, not a search API
    if name == "calculator":
        if set(args) != {"a", "b"} or any(
            type(args[k]) not in (int, float) for k in ("a", "b")
        ):
            raise ValueError("calculator requires two numbers")
        return str(operator.mul(args["a"], args["b"]))
    raise ValueError(f"unknown tool: {name}")
 
def agent(prompt, max_iters=8):
    messages = [{"role": "user", "content": prompt}]
    for _ in range(max_iters):
        resp = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            tools=TOOLS,
            messages=messages,
        )
        if resp.stop_reason == "end_turn":
            return "".join(b.text for b in resp.content if b.type == "text")
 
        if resp.stop_reason != "tool_use":
            raise RuntimeError(f"Incomplete turn: {resp.stop_reason}")
 
        # Append the assistant turn exactly as the model produced it
        messages.append({"role": "assistant", "content": resp.content})
 
        # Run every tool_use block and append tool_results in one user turn
        tool_results = []
        for block in resp.content:
            if block.type == "tool_use":
                try:
                    out = run_tool(block.name, block.input)
                    is_error = False
                except ValueError as error:
                    out = str(error)
                    is_error = True
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": out,
                    "is_error": is_error,
                })
        messages.append({"role": "user", "content": tool_results})
 
    return "[stopped: max_iterations reached]"
 
print(agent("What is 7% of last year's revenue?"))

The Messages API places tool_result blocks in a "role": "user" message, even though the runtime supplies those results. OpenAI Chat Completions uses a separate "tool" role; Responses uses function_call_output items; the shape is different but the idea is identical.

The iteration cap bounds the number of model turns, and run_tool validates this example’s arguments. The [stopped: max_iterations reached] fallback is what you return when the model never chooses to end_turn. In production you'd also wrap run_tool in a timeout and count consecutive errors.

Misconceptions

"The model is the agent." This is common shorthand. The model generates tokens and structured tool requests. The runtime executes those requests and supplies observations to the next generation. The model can then use that evidence to decide what to request next. When Claude Code "decides" to re-run tests after editing a file, what actually happens is the model, prompted with a transcript that contains a failed test run and an edit, generates a tool_use block that calls the test runner.

"More iterations make the agent more capable." A larger cap can help tasks that genuinely need more actions, but also lets a failing trajectory consume more resources. Tune it against task success, cost, and recovery behavior.

"Tool errors should be hidden from the model." For a recoverable failure, return enough sanitized detail to support a different next action. A validation error can identify an argument to fix; a connection error can identify an unavailable service. Remove secrets and sensitive internal details, and stop rather than retry when the failure requires it.

"The loop is just a for-loop." The iteration alone leaves execution permissions, timeouts, error handling, and context assembly unspecified. Those policies determine what the next model turn can do and what evidence it receives.

What's next

The loop needs to preserve tool calls and results in the format expected by the next model request. The next post covers transcript formats and the Harmony spec: provider message schemas, model-facing serialization, and the boundaries a chat template must preserve.

Additional reading (and watching)

• Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. The paper that articulated the think-act-observe loop pattern used by many tool-using systems.

• Anthropic. (2025). Tool use with Claude. Canonical description of the tool_use / tool_result message shape and the agentic loop it implies.

• Shinn, N. et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Studies a separate reflection-and-retry procedure using stored verbal feedback.

• OpenAI. (2025). Responses API and parallel tool calls. Describes how multiple tool calls in one assistant turn are structured and executed concurrently.

• Xu, B. et al. (2023). ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models. Argues for separating the planning phase from tool execution to reduce redundant context traffic, directly motivating context-compaction strategies.

• Anthropic. (2025). Claude Code overview. Describes the tool set and the session-scale loop behavior that powers the CLI.

• OpenAI. (2025). OpenAI Agents SDK. The thin framework wrapping the Responses API loop with handoffs, guardrails, and tracing.

• Model Context Protocol. (2024–2026). MCP specification. The open protocol for tool, resource, and prompt exchange between agent runtimes and external systems.