The Agent Loop: Model, Runtime, Tool, Resume

The word "agent" gets used a lot. The mechanical thing underneath it, the part that makes a language model behave like something with agency, is a loop. A small, four-step loop that the runtime runs, and the model doesn't even know exists.

In the previous post we saw how function calling works as structured generation: the model emits a tool_use block, some serializer makes sure the arguments are well-formed JSON, and you get back a request to run a tool. That's one turn, one call. The API gives you a single chat completion and then stops.

But a 2026 agent has to do something very different. It needs to look up a file, read it, decide to search the web, summarize the result, call a calculator, and only then write its answer. That's not one completion. It's a sequence of completions with tool executions stitched between them, and the thing stitching them together is code you own. That code is the agent loop.

This post is about that loop. What it does, what the model's share is versus the runtime's share, where it breaks in production, and how it ends.

The API only runs once

Call the Anthropic Messages API or the OpenAI Responses API with a user message and a tool schema, and one of three things happens. The model emits a plain text answer. Or it emits a tool_use block and stops, with a stop reason that says "I want to call a tool." Or it hits a max-tokens cap or errors out.

In all three cases, the provider's job is done. The HTTP request returns. If the model asked for a tool, nothing is going to run that tool for you. If the model needs a second turn after seeing the tool's output, nothing is going to ask for that second turn. The provider gave you exactly one forward pass.

So to build anything that feels like "the model used five tools and then answered," you have to be the thing that does the stitching. You detect the stop condition, run the tool, append the result back to the transcript, and call the model again. And again. And again. Until either the model says it's done or your safety rails say it has to be done.

That outer loop is the agent. The model is the token generator inside it.

Here is the handle I want you to walk away with: the model is a turn-taker; the runtime is the conductor. The model has no memory of past iterations other than what the runtime chooses to paste into the next prompt. Everything you think of as "the agent reasoning over time" is really the runtime maintaining a transcript on the model's behalf and handing it back each turn.

Prerequisites

This post assumes the previous two in Arc 10: a short history of agents, which is where the loop's lineage comes from, and function calling as structured generation, which covers how a tool schema turns into a well-formed tool_use block. Who actually runs the code across systems is the subject of MCP, two posts from now; all you need here is that the runtime does. This post also assumes the KV cache and prefix caching material from Arc 5, because the loop eats a lot of tokens and prefix caching is how providers avoid re-encoding the whole transcript every iteration.

If any of those feel fuzzy, go back first. The loop is a lot more intelligible when you can see where the latency and the cost are going.

The minimal loop

Strip away all the production guardrails and the whole idea is thirty lines of pseudocode:

messages = [{role: "user", content: prompt}]
while True:
    response = model.generate(messages, tools=TOOLS)
    if response.stop_reason == "end_turn":
        return response.text
    for block in response.tool_use_blocks:
        result = run_tool(block.name, block.arguments)
        messages.append({role: "assistant", content: response.content})
        messages.append({role: "tool", tool_use_id: block.id, content: result})

That's it. Generate. Check the stop reason. If the model asked for tools, run them, append their results, generate again. Exit when the model signals it's done.

Every production agent loop is a descendant of this. Claude Code's loop, Cursor's, Devin's, LangChain's AgentExecutor, OpenAI's Agents SDK, Anthropic's cookbook examples. They all decorate the same skeleton with retries, timeouts, parallel tool execution, context compaction, and observability hooks, but the spine is the same four states.

The four-state cycle

I find it easiest to keep this loop in my head as a state machine with four nodes, labeled by who is doing the work:

1. GENERATE — the model emits tokens, possibly ending in a tool_use block.
2. PARSE — the runtime reads the stop reason and extracts tool calls.
3. EXECUTE — the tool (your code, a container, an API, whatever) runs and produces a result.
4. RESUME — the runtime appends the result to the transcript and calls the model again.

Four states, three actors. The model generates. The runtime parses and resumes. The tool executes. Nothing else is happening. If you asked "where does 'the agent' think?" the honest answer is: only during GENERATE, and only ever on text it can see in this turn's prompt.

On each GENERATE step, the runtime hands the model the entire conversation-so-far, the model produces one more message, and the loop continues. There's no hidden scratchpad. There's no agent memory module. There's the transcript, and what the model can do with it on its next turn.

What the runtime actually does

The model's share of the work is easy to describe: generate the next message. Everything else belongs to the runtime, and in a real system "everything else" is the overwhelming majority of the code.

Dispatch. When the model emits a tool_use block with name: "search", something has to look up search in a registry of callables and invoke it with the provided arguments. Simple when you have two tools, less simple when you have an MCP server advertising sixty of them and you need to route by namespace.

Argument validation. The provider tries to make tool arguments conform to your JSON schema via constrained decoding, but "tries" is doing work. The runtime should still validate, and it should surface validation failures back to the model as tool errors rather than crashing the loop.

Timeouts and retries. Tool calls are network calls. Network calls hang. The runtime wraps each EXECUTE in a timeout, and on transient errors retries with exponential backoff. When the retries exhaust, the loop doesn't die; it appends {error: "tool X failed after 3 retries"} as the tool result and lets the model decide what to do with that information.

Error surfacing. This one matters a lot. When a tool fails, the right move in almost every case is to hand the error back to the model as if it were a normal tool result. The model then sees the failure in its next GENERATE step and can choose to retry with different arguments, fall back to another tool, or give up and report the failure to the user. Hiding errors from the model turns a recoverable situation into a silent dead end.

Context assembly. The runtime decides which turns go into the next prompt. Every turn? The last N? A summarized version of the first 20? This is the single highest-leverage decision in loop design and we come back to it below.

Parallel execution. When the model emits several tool_use blocks in one turn (Anthropic's Messages API allows multiple; OpenAI's Responses API calls this "parallel tool calls"), the runtime can run them concurrently rather than one at a time. A question like "what's the weather in Tokyo, Paris, and New York?" is three independent calls. Sequential wait is wasted wall-clock.

Parallel execution is one of the clearer gaps between a well-tuned runtime and a naive one. Three independent calls waiting in sequence is just wasted wall-clock.

How a loop ends

Every loop needs stop conditions. There are three that matter, and a runtime missing any one of them will eventually run forever on someone else's dime.

end_turn. The happy path. The model emits a message that is just text, no tool_use blocks, and the stop reason is end_turn. The runtime returns the text to the caller and exits.

max_iterations. The first guardrail. Even a cooperative model can get stuck in "search, read, search again, read again" cycles on queries where no amount of searching will produce the answer. A hard cap on iterations (8, 16, 32, pick your number) means a confused loop fails fast instead of burning a whole context window.

error_threshold. The second guardrail. Three tool failures in a row, or more broadly N failures out of M recent attempts, should trip a circuit breaker. If your database is down, ten retries of the same query won't bring it back, they'll just run up the bill.

There's a useful mental split between these. end_turn is the model's decision. max_iterations and error_threshold are yours. The model can always ask to keep going. The runtime has to occasionally say no.

Context is the budget you actually spend

Tool results take tokens. A lot of them. Fetch a web page and you just added 20,000 tokens to the transcript. Grep a codebase and you added 80,000. Run five of those in a loop and you're trying to fit 300k tokens into the 200k context limit your provider advertises, which means one of three things is about to happen: you truncate something, you summarize something, or you fail.

This is the hydration problem from the inference-engine side of the stack showing up again, just with a different name. The model can only attend to what fits in its window. Everything else has to go somewhere, and the runtime's job is to decide where.

There are a handful of strategies, and production agents usually stack them.

• Summarize earlier turns once they're no longer actively informative. Claude Code does this; the transcript up to a checkpoint gets collapsed into a few hundred tokens of running summary and the original tool results drop out.
• Drop tool results once the model has synthesized them into a claim. If turn 3 fetched a file and turn 4 said "the function is defined on line 42," you can often discard turn 3's raw content entirely.
• Externalize to memory. Write tool outputs to a scratchpad file or a vector store the agent can query on demand, instead of leaving them inline. This is the whole premise of the "retrieval" style of agent.

The point is that context management is not a side concern. It's the central scheduling problem of the loop, and it's the thing that separates a five-turn toy demo from an agent that can run for an hour.

A three-step worked example

Suppose the user asks: "What is 7% of last year's revenue?" The agent loop plays out like this:

Iteration 1. GENERATE → the model, seeing the user's question and a tool schema that includes search and calculator, emits search("annual revenue 2025") as a tool_use. PARSE → runtime pulls the name and arguments out of the response. EXECUTE → runtime calls the search API, gets back "Revenue 2025: $412.0M". RESUME → runtime appends the assistant turn and the tool result to the messages list, calls the model again.

Iteration 2. GENERATE → the model now sees the revenue number. It knows 7% is a multiplication problem and it has a calculator, so it emits calculator(412_000_000 * 0.07). PARSE, EXECUTE, RESUME as before. The result 28_840_000 gets appended.

Iteration 3. GENERATE → the model has the multiplication result. No more tools needed. It emits plain text: "About $28.84M." stop_reason is end_turn. The loop exits. The runtime returns the final text.

Three iterations. Two tool calls. One final answer. The AgentLoop_StateMachine viz above plays this exact trace; you can watch the four-state cycle tick through twice before the third GENERATE terminates cleanly.

Notice what the model never sees: it doesn't know the runtime retried the search once, doesn't know the calculator call took 12 ms, doesn't know anything about the loop's max_iterations setting. All of that is invisible to it. It only ever sees the transcript.

When you stare at one of these runs end-to-end, the intuition flips from "the model is slow" to something more nuanced. GENERATE on iteration 1 is the cold prefill, so yes, that one hurts. Every subsequent GENERATE gets a big discount because the transcript prefix is already in the provider's cache. The fat slice is usually EXECUTE, because EXECUTE is a network call, and network calls are the units of latency that tool loops spend. "Make the agent faster" almost always means "make the tools faster" or "stop shipping bytes the provider has already seen."

Implementation sketch

Here is a minimal loop against the Anthropic Messages API. Roughly forty lines, runnable, no framework. Every production loop is a supersetting of this.

import anthropic, json, operator
 
client = anthropic.Anthropic()
 
TOOLS = [
    {
        "name": "search",
        "description": "Web search",
        "input_schema": {"type": "object", "properties": {"q": {"type": "string"}}, "required": ["q"]},
    },
    {
        "name": "calculator",
        "description": "Multiply two numbers",
        "input_schema": {
            "type": "object",
            "properties": {"a": {"type": "number"}, "b": {"type": "number"}},
            "required": ["a", "b"],
        },
    },
]
 
def run_tool(name, args):
    if name == "search":     return "Revenue 2025: $412.0M"     # stubbed
    if name == "calculator": return str(operator.mul(args["a"], args["b"]))
    return {"error": f"unknown tool: {name}"}
 
def agent(prompt, max_iters=8):
    messages = [{"role": "user", "content": prompt}]
    for _ in range(max_iters):
        resp = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            tools=TOOLS,
            messages=messages,
        )
        if resp.stop_reason == "end_turn":
            return "".join(b.text for b in resp.content if b.type == "text")
 
        # Append the assistant turn exactly as the model produced it
        messages.append({"role": "assistant", "content": resp.content})
 
        # Run every tool_use block and append tool_results in one user turn
        tool_results = []
        for block in resp.content:
            if block.type == "tool_use":
                out = run_tool(block.name, block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": out if isinstance(out, str) else json.dumps(out),
                })
        messages.append({"role": "user", "content": tool_results})
 
    return "[stopped: max_iterations reached]"
 
print(agent("What is 7% of last year's revenue?"))

The "role": "user" on the tool-result turn looks wrong the first time you see it, but it's how the Messages API models tool results: they're formally part of the user's side of the conversation, because the user (i.e. the runtime) is the thing that produced them. The model, on its next turn, sees tool_result blocks in the last user message. OpenAI's API uses a separate "tool" role; the shape is different but the idea is identical.

The for _ in range(max_iters) line is the entire guardrail in this version. The [stopped: max_iterations reached] fallback is what you return when the model never chooses to end_turn. In production you'd also wrap run_tool in a timeout and count consecutive errors, but those are leaves on the same tree.

Misconceptions

"The model is the agent." This is common shorthand. The model is a next-token predictor with a structured-output constraint. It has no memory across turns, no ability to run anything, no way to check whether its last tool call worked. Everything that reads like agency (the sequencing, the memory, the error recovery, the termination) lives in the runtime and in the transcript the runtime builds. When Claude Code "decides" to re-run tests after editing a file, what actually happens is the model, prompted with a transcript that contains a failed test run and an edit, generates a tool_use block that calls the test runner. The decision is a forward pass over a carefully constructed context.

"More iterations make the agent more capable." Past a fairly small point, they don't. The dominant failure mode in agent runs is not "the loop stopped too early" but "the loop wandered." A model that hasn't gotten the answer in 8 iterations usually hasn't framed the problem correctly, and giving it 32 iterations buys you 24 more turns of the same misunderstanding, plus a much fatter context. Most production agents set caps in the 8-16 range and do better with a smart recovery strategy than with more spins.

"Tool errors should be hidden from the model." The instinct is that a stack trace is noise and the model will do something bizarre with it. In practice the opposite is true. Surfacing the error to the model gives it a chance to self-correct. It can read "connection refused" and try a different URL; it can read "invalid JSON at position 47" and fix its arguments. Hiding errors behind a generic "tool failed" message strips exactly the information that would let the model recover. The one place this rule flips is sensitive errors. Don't leak credentials or internal paths, but the fix there is to sanitize the error, not to hide it.

"The loop is just a for-loop." It's shaped like a for-loop, but the hard part is not the iteration; it's the transcript you pass into the next call. Every meaningful difference between a toy agent and a production one lives in context assembly, error surfacing, and stop conditions. The while True is a one-liner, and everything around it is the system.

What's next

This post was the skeleton: four states, three actors, three stop conditions, and context management as the central scheduling problem. What we haven't touched yet is the wire format that carries this transcript between model and runtime. The next post covers transcript formats and the Harmony spec: how a provider-specific messages list gets tokenized into a flat string the model sees, and why getting that tokenization right turns out to be what makes or breaks a chat template.

Additional reading (and watching)

• Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. The paper that articulated the think-act-observe loop pattern that every modern agent runtime descends from.

• Anthropic. (2025). Tool use with Claude. Canonical description of the tool_use / tool_result message shape and the agentic loop it implies.

• Shinn, N. et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Formalized the value of feeding errors back to the model as observations for self-correction within the loop.

• OpenAI. (2025). Responses API and parallel tool calls. Describes how multiple tool calls in one assistant turn are structured and executed concurrently.

• Xu, B. et al. (2023). ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models. Argues for separating the planning phase from tool execution to reduce redundant context traffic, directly motivating context-compaction strategies.

• Anthropic. (2025). Claude Code overview. Describes the tool set and the session-scale loop behavior that powers the CLI.

• OpenAI. (2025). OpenAI Agents SDK. The thin framework wrapping the Responses API loop with handoffs, guardrails, and tracing.

• Model Context Protocol. (2024–2026). MCP specification. The open protocol for tool, resource, and prompt exchange between agent runtimes and external systems.