Function Calling as Structured Generation

A tool definition gives the model a function name, an argument schema, and a description of when to use it. The response can contain a request such as get_weather(city="Tokyo", units="C"). Your application still has to decide whether that request is valid and allowed before it runs anything.

Training teaches the model to choose tools and fill arguments. Constrained decoding, when enabled through a supported strict-output mode, restricts the generated arguments to the schema. Ordinary function calling does not automatically imply that guarantee.

The next post follows the runtime’s execution of the generated call and the return of its result.

The problem

Models output tokens, those tokens get detokenized into text, and text is free-form. A tool call, by contrast, is a highly constrained object: a function name from a known list, a set of named arguments with types, required fields, enums. If any of that goes wrong, the call can't execute, the backend throws, and the agent loop stalls.

You could try to solve this with prompting alone. "Please emit a JSON object with keys city and units, where units is one of C or F." Prompting can produce the requested shape, but does not enforce it. For a hypothetical 95% per-call validity rate, five independent calls are all valid only about 77% of the time. Schemas have corners: nested objects, arrays with typed items, optional fields. Each additional requirement creates another condition the application must validate.

Constrained decoding addresses the formatting part of this problem. At every decoding step, ask: given the partial output and the supported schema, which tokens can legally come next? Mask out the rest, then sample from what survives. Truncation, refusals, and transport failures still need separate handling.

Prereqs

This uses three earlier topics:

• Decoding as sampling from a next-token distribution. Logits, softmax, greedy vs. top-k vs. nucleus. See sampling strategies in Arc 5.
• Chat templates and assistant turns. The tool call gets emitted inside an assistant turn with some wrapper tokens around it. The special tokens and chat templates post from arc 3 covers that boundary.
• Tokenizers don't respect JSON. A single BPE token can straddle a quote and a letter (like "Alice), so the mask must check the whole token’s decoded content. The BPE post has the background.

The one-line version

Let $P(t \mid \text{ctx})$ be the model's next-token distribution given the current context, and let $\text{valid}(s)$ be the set of tokens that keep the output on a path to a schema-valid object, given the current state $s$ of a state machine tracking the partial output. Then the runtime samples from:

In words: zero out the probabilities of every token the schema forbids, renormalize what's left, and sample from that. Assume at least one allowed candidate has positive probability mass. The model's preferences still determine the choice among valid tokens. The schema determines the set of allowed tokens.

A grammar recognizer tracks the partial output to determine $\text{valid}(s)$.

Compiling the schema to a state machine

A JSON schema describes a completed object. A decoder needs to track a partial object. For a small, fixed structure, a finite-state machine (FSM) is a useful picture. General recursive schemas need a richer grammar mechanism; an FSM with finitely many states cannot track arbitrarily deep nesting.

Consider a schema with two required fields: city (a string) and units (an enum of C or F). The diagram shows one accepted path through this fixed structure.

{
  "type": "object",
  "properties": {
    "city": {
      "type": "string"
    },
    "units": {
      "enum": [
        "C",
        "F"
      ]
    }
  },
  "required": [
    "city",
    "units"
  ],
  "additionalProperties": false
}

The FSM has a state for every meaningful position in the evolving output: "just opened a brace," "in the middle of a key string," "about to emit a colon," "expecting a value of type string," "expecting a value from this specific enum." Each state carries a set of allowed next-token classes (open quote, close quote, any alphanumeric, one of C or F, and so on). The runtime holds the current state, asks the model for logits, masks everything outside valid(state) to negative infinity, samples, then advances the state.

Real implementations must also handle token boundaries, JSON escaping, alternate paths, and the supported subset of JSON Schema. Libraries such as Outlines and XGrammar compile and cache constraints, but their internal representations and performance differ.

What constrained decoding actually looks like

At one decoding step, compare the model’s candidate probabilities before and after applying the mask.

(empty prefix)

At the first step in this toy vocabulary, Sure has the largest unmasked weight. Only the chunk {"name": is allowed. At the next step, both "Alice" and "Bob" remain valid, so normalization preserves their relative weights.

Constraint compilation and per-token mask application add work. Caching and optimized grammar engines can reduce that overhead, while restrictive grammars can also reduce the number of generated tokens. Measure the chosen backend, schema, model, and workload.

Why training still matters

A completed output can satisfy the grammar while requesting the wrong function or filling its arguments with the wrong values. The model must also decide whether a tool is needed at all. Training, prompting, and the supplied context influence those choices.

Toolformer and Gorilla explored different ways to teach API use. A modern supervised dataset can include tool definitions, requests, calls, observations, and final answers. Assistant targets can include both the call and the continuation.

It helps to separate the checks rather than collapse them into one reliability number:

{"name":"search","arguments":{"query":"weather in Oslo"}}

Parallel tool calls

A response can contain several tool calls. For a weather query about Tokyo, Paris, and New York, those calls are independent: the runtime can execute them concurrently and return their results together.

APIs represent multiple calls as multiple structured items or blocks in a response. That does not by itself make execution concurrent: your runtime must identify independent calls and schedule them. Strict-schema support and parallel-call restrictions depend on the provider and model.

If three independent tools each take an assumed 200 ms, their tool-execution time is 600 ms in sequence or about 200 ms together, ignoring dispatch overhead. Total request latency also includes generating the calls and the answer. The chart includes those phases under a different explicit set of assumed durations.

A runnable sketch

This Anthropic SDK example defines one tool, sends a request, and inspects the returned tool_use block. OpenAI and Gemini use different response envelopes.

import anthropic
 
client = anthropic.Anthropic()
 
# 1. Declare a tool; this ordinary call does not enable strict mode.
tools = [
    {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "input_schema": {
            "type": "object",
            "properties": {
                "city":  {"type": "string"},
                "units": {"type": "string", "enum": ["C", "F"]},
            },
            "required": ["city", "units"],
        },
    }
]
 
# 2. Ask the model a question. Validate returned inputs before execution.
msg = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=512,
    tools=tools,
    messages=[{"role": "user", "content": "What's the weather in Tokyo in C?"}],
)
 
# 3. Inspect the structured output. No json.loads needed — the SDK
#    hands you a tool_use block with parsed input already.
for block in msg.content:
    if block.type == "tool_use":
        print(block.name)   # "get_weather"
        print(block.input)  # {"city": "Tokyo", "units": "C"}
    elif block.type == "text":
        print(block.text)
 
# 4. In a real agent loop you would now actually call get_weather(),
#    append the result as a tool_result block, and resume the model.
#    That's the next post.

The Anthropic SDK exposes the input as a parsed object. Parsing and schema validity are separate checks; ordinary tool use can still return arguments that fail your validator. For guaranteed structural conformance, use a supported strict tool mode and its schema restrictions. Even then, check permissions and application constraints.

Misconceptions

"All function calls use constrained decoding." Tool-trained models can generate structured calls without a grammar constraint. A provider’s strict mode adds a format guarantee within its documented limits; declaring a tool alone is not a portable way to request that guarantee.

"Constrained decoding is free." Compilation, tokenizer-aware matching, and per-step masking cost work. Optimized backends cache this work where possible. Compare latency and throughput on your actual schema rather than assuming a fixed percentage overhead.

"Function calling means the runtime runs the function." No. Function calling, in the API sense, only covers the structured generation half: the model emits a call, and the SDK hands you a parsed object. The actual execution is on you (or on a server-side "built-in tool" if the provider offers one). The executor can be a Python function, an MCP server, or a provider-hosted code interpreter. The next post picks up exactly there.

"A valid JSON object is a correct tool call." A completed strict output can meet supported syntax and schema constraints. It does not give you semantic correctness. The model can still pick the wrong tool, pass a schema-valid "Paris" when the user requested Tokyo, or decide to call the weather tool when the user was asking about the stock market. Application checks and evaluation must cover those errors separately.

What's next

After a call is generated, the runtime must validate it, execute it, and return the result. The next post follows The Agent Loop: Model, Runtime, Tool, Resume, including how to handle failures and decide when to stop.

Additional reading (and watching)

• Willard, B. T., & Louf, R. (2023). Efficient Guided Generation for Large Language Models. arXiv:2307.09702. Introduces the FSM-based grammar-guided decoding approach used in early Outlines work. The renormalization equation in this post is from their formulation.

• Outlines (Willard et al., 2023–). Reference implementation of FSM-based structured generation for LLMs. An implementation of constrained generation; backend selection and APIs vary by engine version.

• Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761. A method for training on useful tool-call annotations; external code still executes the tools.

• Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334. Studied fine-tuning for API-call generation; the Gorilla team later launched the Berkeley Function Calling Leaderboard.

• Yan, F., Mao, H., Ji, C. C.-J., et al. (2024). Berkeley Function Calling Leaderboard V2. Gorilla blog. Version-specific evaluation results on real and live tool-use benchmarks, separated by parseability, selection, and argument correctness.

• Anthropic. (2024). Tool use with Claude. Covers the tool_use block format, parallel tool calls, and the interaction with chat templates. OpenAI's equivalent is the Function Calling guide.

• Gerganov, G., et al. (2023–). llama.cpp GBNF grammars. Documentation for the GBNF grammar format and sampler-level integration. Constrains local-model decoding with supported grammars, including supported JSON Schema conversions.

• OpenAI. (2024). Introducing Structured Outputs in the API. Announcement and technical notes for OpenAI's grammar-constrained JSON output mode, which guarantees schema-valid responses at the API layer.

• JSON Schema Organization. (2020). JSON Schema 2020-12. The schema language behind many tool definitions; providers support different subsets.

• Anthropic. Strict tool use. Separates ordinary tool use from schema-enforced arguments.