Tool-Use Fine-Tuning

A tool-use trajectory is just a four-role conversation — user, assistant tool-call, tool result, assistant answer — flattened into one token stream and SFT-trained with only the assistant tokens scored.

A common first attempt at tool use looks like this. Write a system prompt that says "respond with JSON like this," give a couple of examples, and send a request. It works for a while. Then it stops working. The model calls a function that doesn't exist, puts a string where an integer belonged, or wraps the JSON in a sentence like "Here is the tool call you requested:" which breaks the parser.

A base model has no reason to emit rigid JSON. Nothing in its pretraining tells it that the space after a comma matters, or that hallucinating a plausible-looking function name is the worst possible failure mode. Pretraining teaches it to imitate the distribution of text on the internet, and most of the internet isn't function calls.

So reliable tool use turned out to be a training problem, not a prompting problem or a decoding trick. That's what this post is about.

Prerequisites

This post assumes you've read:

• Supervised Fine-Tuning for how SFT reshapes a pretrained base model with (prompt, response) pairs, and the loss-masking convention where only the response tokens contribute to the gradient.
• Special Tokens and Chat Templates for how a multi-turn conversation gets flattened into one big token sequence with role markers. That format is the spine of everything below.

The problem

Let me sharpen it. Say you have a pretrained base model and a function:

get_weather(location: string) -> { temp_c: number, condition: string }

You want the model, when a user asks "what's the weather in Tokyo," to emit exactly this text:

{ "name": "get_weather", "args": { "location": "Tokyo" } }

Not a sentence explaining what it's going to do. Not a markdown code block. Not get_weather("Tokyo") in Python syntax. Not an imagined get_current_weather with a slightly different argument name. The exact string above, character for character. Your calling code needs to json.loads it, dispatch to the right function, and get a result back.

Prompting a base model to do this reliably is a losing game. You can get it to 70%-ish with good examples and careful phrasing. The rest of the time it does something creative, and each creative failure is a different creative failure, so you can't patch them all at the prompt level.

The numbers above are representative of what's been reported in the tool-use literature. Your exact rates will depend on the base model, the tool surface, and how brittle the check is. The shape holds across every public benchmark I've seen.

Now. The most important thing to notice here is that the four bars aren't measuring the same thing. "Valid JSON" is the easiest bar to clear, because there are lots of valid JSON strings. "Arg types match schema" is the hardest, because it's a conjunction of constraints. Fine-tuning lifts all four bars together, which tells me the model is learning the whole contract rather than just picking up a better sense of when to emit braces.

What "training it" actually means

Tool-use fine-tuning is SFT. Same loss, same optimizer, same loss-masking rule. The only thing that changes is what lives inside the training examples.

A plain SFT example is a (prompt, response) pair:

prompt: "Write a haiku about the sea."
response: "Silent tide breathes in..."

A tool-use SFT example is a full trajectory: the user's request, the assistant's tool call, the tool's response, and the assistant's final answer. The whole sequence gets flattened through the chat template (the same one from Special Tokens and Chat Templates) into one long stream of tokens. Role markers like <|assistant|>, <|tool_call|>, and <|tool|> tell the model, at every moment, what role is "speaking." Then the loss is computed over the assistant's outputs only. Everything the user said, everything the tool returned, is masked. The gradient flows only through tokens the model itself would be expected to generate.

Watch the loss bars as the sweep moves across. Every assistant token counts: the tool call's opening {, the "name" field, the quoted argument values, the closing }, and the final answer at the end. Everything else is grey. This is exactly the loss-masking rule from the SFT post, extended to a four-role conversation instead of two.

Here's the thing that makes this work. The JSON structure is not imposed from outside. It's just tokens in the target sequence, like any other tokens. The model doesn't know it's "JSON"; it knows the next-token probability of " is very high after a certain pattern, and very low otherwise. After enough examples, the grammar lives in the weights.

Minimal math

It's genuinely the SFT loss, no trick. If the training sequence is $t_1, t_2, \dots, t_n$ and $M \subseteq \{1, \dots, n\}$ is the set of positions on assistant tokens (including the tool-call tokens), then the loss is:

Same causal conditioning as autoregressive language modeling. Same cross-entropy-with-a-one-hot-target as every other SFT step. The only thing that matters is which positions are in $M$. You add positions that live inside the JSON of the tool call, and the gradient starts pushing the model's distribution toward that exact sequence of quotes, commas, colons, braces, and field names.

Tool results, the environment's reply, are also in the training sequence, because the model needs to condition on them to write the final answer. But they are not in $M$. You don't want the model's gradients to memorize tool outputs as if it had to generate them itself; you want the model to treat them as input it reads. Masking the tool-result tokens out of the loss is what draws that line.

The role boundaries stop being visually obvious once the chat template crams everything into one token stream. But the loss mask is where the line lives.

A worked trajectory

Let me trace through one end-to-end example. This is essentially the trajectory the hero loop is showing, but slowed down with explicit token-level steps.

The user sends: "What's the weather in Tokyo?"

The full training example, in logical form:

system:     "You have access to a get_weather tool."
user:       "What's the weather in Tokyo?"
assistant:  <tool_call>{"name":"get_weather","args":{"location":"Tokyo"}}</tool_call>
tool:       {"temp_c": 18, "condition": "cloudy"}
assistant:  "It's 18 C and cloudy in Tokyo right now."

Now we flatten this through the chat template. Llama 3 uses <|start_header_id|>...<|end_header_id|> around a role name, then the content, then <|eot_id|>. ChatML uses <|im_start|>role ... <|im_end|>. Different tokens, same idea. The model sees one long token sequence.

When we compute the loss, every token in the two assistant segments is scored. The system and user and tool segments are masked out. That includes the raw JSON inside the tool call. Yes, the quotes, the braces, the field names. All of that is just "tokens the assistant emits," so all of that is scored.

After a pass through a training dataset of a few thousand trajectories like this one, something clicks. The model has seen enough tool calls that the token-level transitions (after <|start_tool_call|>, the next token is almost certainly {; after {, it's almost certainly ") become part of its sampling distribution. It now just does that, on new tools it's never seen before, as long as they show up in the system prompt.

Where the data comes from

Thousands of high-quality trajectories is the easy part to say and the hard part to get.

Hand-writing them doesn't scale. Logging real users doesn't work for a model nobody is using yet. The dominant answer in the literature is synthetic data: use a stronger model (a "teacher") to generate trajectories for a weaker model (a "student") to train on. This is the recipe in Toolformer, Gorilla, ToolLLM, and ToolAlpaca, with variations. The core loop is the same across all of them.

A few notes on how this actually works in practice:

The teacher has to be good at the task. If the teacher can't reliably emit valid tool calls, you're distilling noise. A lot of the early difficulty in this space was that the strongest available teacher wasn't much better at tool use than the student you were trying to train. GPT-4-class models changed the calculus here because they were actually reliable enough to be useful teachers.

The tool needs to execute, or at least be simulated. Toolformer executes real APIs during data synthesis and keeps trajectories whose results reduced the loss on a held-out continuation. ToolAlpaca uses a simulated environment. ToolLLM gets 16,000+ real APIs from RapidAPI and executes them. The execution step is what keeps the data grounded; without it, the teacher hallucinates fake API responses and the student learns to expect them.

Filtering is load-bearing. Raw synthesis output is maybe 60-80% usable. The rest is malformed JSON, wrong tool selection, or answers that don't actually use the tool result. A validator filter (parse the call, execute it, check the answer cites the result) is cheap compared to the cost of one bad gradient update.

Diversity matters more than volume. Zhou et al. showed that even 1,000 carefully curated SFT examples can compete with larger noisy datasets in the general case. The same holds for tool use: a small, diverse dataset that covers the tool surface evenly beats a huge one that over-indexes on three easy tools.

So. Synthetic trajectories plus light filtering, a few thousand examples, one training run. That's the recipe.

Parallel tool calls

There's one extra trick that matters. Imagine a user asks "what's the weather in Tokyo, Paris, and New York?" A model trained on single-call trajectories will pick one city, wait for the answer, pick the next, wait again, and so on. Three back-and-forths, three serial network hops. That's slow, both in tokens generated and in wall-clock time.

The fix is to train the model to emit multiple tool calls in a single assistant turn. The training example's assistant turn now contains an array of tool calls, or a sequence of tool-call blocks, and the tool segment that follows contains all the results. From the caller's perspective, you dispatch the three calls in parallel and feed all three results back at once.

This is a specific thing you have to put in the training data. Models don't emit parallel calls by accident. The training-data generator has to include examples like "here's a user request that naturally decomposes into three independent calls, and here's what the parallel call looks like." The teacher has to emit them. The chat template has to have a place to put them. Only then does the student learn the pattern.

OpenAI's function-calling API added parallel calls in November 2023; Anthropic's Claude 3 series supported them from launch. The reason they went from "not a thing" to "standard" in about a year is that it's a training data question, and the data is cheap to generate once you know to do it.

Implementation sketch

Here is a minimal training JSONL entry. One line per example. This is the Llama 3 tool-use flavor; ChatML and Qwen and Mistral look similar, with different special tokens.

{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant. You have access to: get_weather(location: string) -> { temp_c: number, condition: string }."
    },
    {
      "role": "user",
      "content": "What's the weather in Tokyo?"
    },
    {
      "role": "assistant",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {
            "name": "get_weather",
            "arguments": "{\"location\": \"Tokyo\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": "{\"temp_c\": 18, \"condition\": \"cloudy\"}"
    },
    {
      "role": "assistant",
      "content": "It's 18 C and cloudy in Tokyo right now."
    }
  ],
  "loss_mask": {
    "system": false,
    "user": false,
    "tool": false,
    "assistant": true
  }
}

A training loader reads this, applies the tokenizer's chat template to flatten messages into a single token sequence, builds a 0/1 mask the same length as that sequence using loss_mask to mark which role segments are scored, and hands both to the trainer. From there, the training loop is literally the SFT loop from the SFT post. No new machinery.

One thing I want to flag. The function.arguments field is a JSON string, not a JSON object. This is an artifact of the OpenAI function-calling API and it leaks into almost every dataset format. The reason is that the arguments are structured data the model has to emit as text, and keeping them as an escaped string makes the round-trip less ambiguous. It looks weird, but it's intentional.

Misconceptions

"Constrained decoding makes fine-tuning unnecessary." Constrained decoding, which we cover in Structured Generation and Function Calling, forces the decoder to only sample tokens that keep a JSON grammar valid. That's good. It guarantees you'll get parseable JSON even from a weak model. But it can't make the model pick the right tool, or fill in the right arguments, or know when to stop calling tools. Those are semantic choices the model has to learn. Constrained decoding is the bottom layer of the safety net; fine-tuning is the whole stack above it. They're complementary.

"If the tool isn't in the system prompt, the fine-tuned model won't call it." Mostly false, which is the interesting part. Fine-tuning on a diverse set of tools teaches the model a general skill: read the tool spec in the system prompt, emit a call that matches that spec. A well fine-tuned model can call a tool it's never seen before at training time, as long as the spec is in context. That's the whole reason tool use is a useful capability rather than a per-tool specialization.

"The training data needs hundreds of thousands of examples." A few thousand diverse, high-quality trajectories is often enough for a small-to-medium model, echoing the LIMA finding. The returns diminish fast beyond that point, and the dataset can actively start to hurt if it over-represents a few tool patterns and the model collapses onto them. Diversity beats scale here.

What's next

The next post covers long-context training techniques, where the problem shifts from "what does the model emit" to "how does the model handle inputs that are 10x or 100x longer than anything it saw in pretraining."

Additional reading (and watching)

• Patil, S. G., et al. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334. Shows that even a small fine-tune on API-calling trajectories closes most of the reliability gap to GPT-4 on API-selection tasks.

• Yan, F., et al. (2024). Berkeley Function Calling Leaderboard. UC Berkeley Gorilla team. The standing leaderboard for tool-calling reliability across models, with breakdowns by tool category and failure mode.

• Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS 2023. The paper that crystallized the self-supervised recipe for tool-use training: insert candidate tool calls, execute them, keep the ones that reduce downstream loss.

• Patil, S. G., et al. (2023). Gorilla. arXiv:2305.15334. Fine-tunes LLaMA on synthetic API-calling trajectories covering HuggingFace, TensorHub, and TorchHub APIs. First widely-cited demonstration that a 7B open-weight model can compete with GPT-4 on API selection.

• Qin, Y., et al. (2023). ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv:2307.16789. Demonstrates that fine-tuning generalizes to tools unseen at training time, given a sufficiently diverse catalog.

• Tang, Q., et al. (2023). ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases. arXiv:2306.05301. Focuses on the synthetic-trajectory pipeline in a simulated tool environment, shows the LIMA-style data-quality effect holds for tool use.

• Zhou, C., et al. (2023). LIMA: Less Is More for Alignment. NeurIPS 2023. The "1,000 curated examples can compete with 50k noisy ones" result that the tool-use community repeatedly confirmed.

• Anthropic. (2024). Tool use with Claude. Anthropic documentation. Describes the tools / tool_use / tool_result message shapes, parallel tool-call behavior, and the underlying training contract from the API side.

• OpenAI. (2023). Function calling and other API updates. OpenAI blog. The announcement that productized tool-use fine-tuning as a provider API and shaped the JSONL dataset conventions that most open-weight models later adopted.

• Meta AI. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. Section on tool use describes the training mixture, the special tokens, and the zero-shot / few-shot evaluation protocol. The source for the public Llama 3 tool-use template.

• Yan, F., et al. (2024). BFCL: Berkeley Function Calling Leaderboard v3. Expanded evaluation covering nested, parallel, and multi-turn tool use, the closest thing to a standard benchmark for "real" tool-use reliability in 2026.

• Liu, X., et al. (2023). AgentBench: Evaluating LLMs as Agents. ICLR 2024. Broader agent evaluation that includes tool-use subtasks and has become a de facto reference for the "does the model actually complete the task" end of the reliability question.