Function Calling as Structured Generation

When you call a modern model API with a tool definition, something interesting happens. You hand the model a JSON schema for a function, you ask it a question, and back comes a perfectly-formed JSON object that matches the schema. Keys in the right places, strings quoted, numbers unquoted, enums respecting their allowed values. You can json.loads the thing and hand it straight to your backend.

The obvious story is that the model has gotten really good at emitting JSON. That story is a little bit true and mostly wrong. Function calling is really structured generation: at decode time, the runtime is masking the model's output distribution to only ever allow tokens that keep the growing string on a path to a valid object against your schema. The model proposes, and a finite-state machine disposes.

Arc 10 is about tools, protocols, and agent loops. "Function calling" gets used to mean two different things worth keeping separate: the structured generation half, where a model emits a well-formed tool call, and the tool execution half, where the runtime actually runs the function and feeds the result back. This post is about the first half only. The next post is about the second.

The problem

Models output tokens, those tokens get detokenized into text, and text is free-form. A tool call, by contrast, is a highly constrained object: a function name from a known list, a set of named arguments with types, required fields, enums. If any of that goes wrong, the call can't execute, the backend throws, and the agent loop stalls.

You could try to solve this with prompting alone. "Please emit a JSON object with keys city and units, where units is one of C or F." A strong model will comply most of the time, and people shipped exactly this kind of thing in 2023. But "most of the time" is the wrong reliability target for production. A 95% valid-JSON rate at one tool call per turn becomes 77% reliability across five turns. Schemas have corners: nested objects, arrays with typed items, optional fields. Free-form generation leaks at every corner.

The fix that made tool use production-ready is to stop trusting the model to emit valid JSON and start forcing it to. At every decoding step, ask: given the string we've emitted so far and the schema the caller gave us, which tokens could legally come next? Mask out the rest, and sample only from what survives.

Prereqs

I'm going to assume you're comfortable with a few things from earlier arcs:

• Decoding as sampling from a next-token distribution. Logits, softmax, greedy vs. top-k vs. nucleus. If that's cloudy, the sampling strategies post in Arc 5 is the right refresher.
• Chat templates and assistant turns. The tool call gets emitted inside an assistant turn with some wrapper tokens around it. The special tokens and chat templates post from arc 3 covers that boundary.
• Tokenizers don't respect JSON. A single BPE token can straddle a quote and a letter (like "Alice), which turns out to matter a lot for how the masking is implemented. The BPE post has the background.

Everything else I'll build up in-line.

The one-line version

At the heart of structured generation is a single equation. Let $P(t \mid \text{ctx})$ be the model's next-token distribution given the current context, and let $\text{valid}(s)$ be the set of tokens that keep the output on a path to a schema-valid object, given the current state $s$ of a state machine tracking the partial output. Then the runtime samples from:

In words: zero out the probabilities of every token the schema forbids, renormalize what's left, and sample from that. The model's preferences still determine the choice among valid tokens. The schema determines the set of allowed tokens. Those two knobs are orthogonal.

The interesting part is $\text{valid}(s)$. That's where the finite-state machine lives.

Compiling the schema to a state machine

A JSON schema is a static description. A stream of tokens is a temporal thing. You can't mask tokens directly against "the schema"; you need something that tracks, moment by moment, where you are in the schema. That something is a finite-state machine.

Compiling a JSON schema to an FSM means walking the schema and emitting transitions. Here's the idea on a tiny example: a schema with two required fields, city (a string) and units (an enum of C or F).

The FSM has a state for every meaningful position in the evolving output: "just opened a brace," "in the middle of a key string," "about to emit a colon," "expecting a value of type string," "expecting a value from this specific enum." Each state carries a set of allowed next-token classes (open quote, close quote, any alphanumeric, one of C or F, and so on). The runtime holds the current state, asks the model for logits, masks everything outside valid(state) to negative infinity, samples, then advances the state.

The real implementation is fussier than this suggests. Real FSMs have thousands of states, not ten, because they track every schema path and every tokenizer quirk. The Outlines library, from Willard and Louf, does this compilation ahead of time and caches the result per-schema, which is what makes structured generation fast enough to run in production.

What constrained decoding actually looks like

Let me make this concrete. Here's the same idea from the other direction: fix a specific point in the decoding, look at what the model wanted to emit, and look at what the mask let through.

The token the model would have emitted without the mask is often reasonable. At the opening state, it wants to emit Sure or Here, which is perfectly natural prose. The mask forces it to emit { instead. At the COLON state, it wants to emit a quote, but a colon is the only thing that's legal.

The model isn't being wrong. It's doing what language models do: producing a distribution over plausible next tokens given the context. Structured generation is an external constraint on top of that distribution, and the mask is where the "function calling" happens. Without it, even a good model will drift into prose somewhere around the third nested object.

The cost is real but usually small. Every decode step has to run the mask (checking each token ID against the FSM state) and renormalize. Modern implementations precompute per-state token ID bitmasks, which turns the per-step overhead into a handful of bitwise operations. On vLLM with Outlines, the throughput hit for structured generation is typically 10-20%, not 2×. You get a hard guarantee in exchange: the output parses, every time.

Why training still matters

You might be wondering: if the FSM guarantees valid JSON, why do models need to be trained on function calling at all? Wouldn't any model, constrained by a strong enough grammar, produce correct tool calls?

The answer is that a schema-valid tool call is not the same thing as a correct tool call. The FSM guarantees the output parses. It doesn't guarantee the model picked the right function, filled in the right arguments, or even decided that a tool call was the right move for this turn. All of that is judgment, and judgment comes from training.

The recipe, pioneered by papers like Toolformer and then formalized in the Gorilla line of work, is to construct a dataset of (tools, user_request, tool_call, tool_result, final_answer) quintuples and fine-tune the base model on them. The model learns three things at once: when a tool call is warranted, which tool to pick, and how to fill in the arguments. The JSON-emission mechanics get reinforced along the way, but the bigger lift is the judgment piece.

What this buys you, compared to prompting alone, is visible in the reliability numbers:

This is the moment where the two halves of the story lock together. Constrained decoding handles the mechanics: every byte that comes out will parse, every enum value will be from the allowed set, every required field will be present. Training handles the semantics: the model picks the right tool and fills in the right arguments. Neither half works alone. A prompted-only model with no grammar hits maybe 60-75% end-to-end reliability. A fine-tuned model without grammar hits maybe 90-95%. Put them together and you're reliably above 99% on well-specified schemas, which is the floor below which agent loops start breaking in ugly ways.

Parallel tool calls

Once you've shipped structured generation, a natural extension is to let the model emit multiple tool calls in a single turn. If the user asks for the weather in Tokyo, Paris, and New York, you don't want three serial round trips. You want the model to emit all three calls at once, the runtime to fan them out concurrently, and the results to come back as a batch.

Mechanically, parallel tool calls are a small extension of the FSM. The schema for the model's output becomes "an array of tool-call objects" instead of "one tool-call object." The FSM gets an outer loop that accepts a comma-separated list. Everything else is the same: each inner call is still masked against its own tool schema, the model still picks the arguments, the wrapper tokens still demarcate the block.

The win is latency. A sequential three-call trajectory with 200ms per tool pays ~600ms of wall-clock that a parallel trajectory collapses to ~200ms. For agent loops that make many tool calls per turn (web search plus a database query plus a calendar check), the difference is a pleasant UX versus a frustrating one. All major providers expose this now; OpenAI calls it "parallel function calling," Anthropic lets you emit multiple tool_use blocks in one assistant turn, and Gemini supports it through function-call arrays.

A runnable sketch

Here's the smallest end-to-end example I can write. It defines a single-tool schema, calls a model with it, and inspects the structured output. I'm using the Anthropic SDK because tool_use blocks make the structure explicit, but the OpenAI and Gemini versions look nearly identical.

import anthropic
 
client = anthropic.Anthropic()
 
# 1. The tool schema — this is what gets compiled to an FSM server-side.
tools = [
    {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "input_schema": {
            "type": "object",
            "properties": {
                "city":  {"type": "string"},
                "units": {"type": "string", "enum": ["C", "F"]},
            },
            "required": ["city", "units"],
        },
    }
]
 
# 2. Ask the model a question. Under the hood, the runtime will
#    mask decoding against the tool schema if the model opens a
#    tool_use block.
msg = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=512,
    tools=tools,
    messages=[{"role": "user", "content": "What's the weather in Tokyo in C?"}],
)
 
# 3. Inspect the structured output. No json.loads needed — the SDK
#    hands you a tool_use block with parsed input already.
for block in msg.content:
    if block.type == "tool_use":
        print(block.name)   # "get_weather"
        print(block.input)  # {"city": "Tokyo", "units": "C"}
    elif block.type == "text":
        print(block.text)
 
# 4. In a real agent loop you would now actually call get_weather(),
#    append the result as a tool_result block, and resume the model.
#    That's the next post.

First, there's no json.loads call anywhere. The SDK parses for you because the server already guaranteed the bytes are valid JSON against your schema. Second, the schema goes in once at the top, and the model, the FSM, and the SDK all agree on its shape. Changing the schema is the only place the structure lives.

Misconceptions

"The model emits JSON because it's smart." It emits JSON because it's been trained to produce tool-call-shaped output and the runtime is masking its sampling against a grammar derived from your schema. Take either half away and the reliability drops. Take both away and you're back in the 2023 world of "please reply in valid JSON" and a ~95% valid-JSON rate. Function calling is two layers stacked on top of each other: training makes the model want to emit a tool call, and constrained decoding makes sure what it emits actually parses.

"Constrained decoding is free." It isn't. Every decode step incurs a mask computation against the current FSM state, which for a large vocabulary is not trivial. Modern implementations are clever about it (precompiling per-state token bitmasks, caching FSMs per schema, merging contiguous allowed ranges) and the overhead usually lands in the 10-20% range on a well-tuned inference engine. On a naive implementation, especially for deeply nested schemas with many optional branches, you can easily see a 2-3× slowdown. The engineering matters.

"Function calling means the runtime runs the function." No. Function calling, in the API sense, only covers the structured generation half: the model emits a call, and the SDK hands you a parsed object. The actual execution is on you (or on a server-side "built-in tool" if the provider offers one). This distinction matters because it's what makes tool loops compositional: the same structured-generation primitive serves your Python function, an MCP server, and a provider-hosted code interpreter. The next post picks up exactly there.

"A valid JSON object is a correct tool call." Constrained decoding gives you parseability and schema-validity. It does not give you semantic correctness. The model can still pick the wrong tool, pass "Tokyo, Japan" where the schema expected "Tokyo", or decide to call the weather tool when the user was asking about the stock market. Those are training and prompting problems. Don't conflate "it parsed" with "it's right."

What's next

Structured generation is the first half of the tool-use story: the part where a model emits a well-formed call. The second half is the agent loop: what happens after the call is emitted, how the runtime executes the function, feeds the result back into the model's context, and lets the model decide whether to call another tool or answer. That's where tool use becomes agentic and where the real systems-design work lives.

The next post covers The Agent Loop: Model, Runtime, Tool, Resume, the four-step cycle that turns a single tool call into an agent, and the handful of design choices that determine whether your agent reliably gets work done or gets stuck in a loop calling search fourteen times.

Additional reading (and watching)

• Willard, B. T., & Louf, R. (2023). Efficient Guided Generation for Large Language Models. arXiv:2307.09702. Introduces the FSM-based grammar-guided decoding approach that underlies Outlines and, by extension, most production structured-generation stacks. The renormalization equation in this post is from their formulation.

• Outlines (Willard et al., 2023–). Reference implementation of FSM-based structured generation for LLMs. Integrated as the default backend for vLLM's guided_json and guided_grammar parameters.

• Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761. Early demonstration that tool use could be learned end-to-end rather than orchestrated by external scaffolding. Not the direct ancestor of modern function calling but an important conceptual stepping stone.

• Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334. Established the modern recipe of fine-tuning on tool-use trajectories; the Gorilla team later launched the Berkeley Function Calling Leaderboard.

• Yan, F., Mao, H., Ji, C. C.-J., et al. (2024). Berkeley Function Calling Leaderboard V2. Gorilla blog. Current-frontier reliability numbers on real and live tool-use benchmarks, separated by parseability, selection, and argument correctness.

• Anthropic. (2024). Tool use with Claude. Covers the tool_use block format, parallel tool calls, and the interaction with chat templates. OpenAI's equivalent is the Function Calling guide.

• Gerganov, G., et al. (2023–). llama.cpp GBNF grammars. Documentation for the GBNF grammar format and sampler-level integration. Lets any CFG constrain local-model decoding, including JSON schemas translated to BNF.

• OpenAI. (2024). Introducing Structured Outputs in the API. Announcement and technical notes for OpenAI's grammar-constrained JSON output mode, which guarantees schema-valid responses at the API layer.

• JSON Schema Organization. (2020). JSON Schema 2020-12. The spec that defines the schema language every major provider uses as the input to their FSM compilers. Worth skimming once even if you never read another spec.