Skip to content
Arc 7: Training & Post-Training

Tool-Use Fine-Tuning

7 min readbuilds on Supervised Fine-Tuning, Special Tokens, Chat Templates, and Input Formatting and Structured and Constrained Generation
One toy tool exchange. Select a role to pause on that message.
  1. What's the weather in Tokyo?
  2. {
  "name": "get_weather",
  "args": {
    "location": "Tokyo"
  }
}
  3. {
  "temp_c": 18,
  "condition": "cloudy"
}
  4. It's 18°C and cloudy in Tokyo.

Schematic payloads; the runtime executes the requested tool and returns its result.

This example serializes four messages: a user request, an assistant tool call, a tool result, and an assistant answer. Its SFT mask scores only assistant targets.

A prompt can demonstrate a tool-call format, yet a model may still invent a function, emit an argument with the wrong type, or add prose that breaks the parser.

Pretraining can include JSON, source code, and API calls, but that does not make a model reliable at choosing and using your tools. A useful training example connects the request, the available tool schema, the call, and the result.

Fine-tuning can improve those choices. Prompting supplies the available tools and task instructions; constrained decoding can enforce an output schema. Training, decoding, and caller-side validation address different parts of the problem.

Prerequisites

This post assumes you've read:

  • Supervised Fine-Tuning for how SFT reshapes a pretrained base model with (prompt, response) pairs, and the loss-masking convention where only response targets contribute directly to the loss.
  • Special Tokens and Chat Templates for how a multi-turn conversation gets flattened into one big token sequence with role markers.

The problem

Suppose you have a pretrained base model and a function:

get_weather(location: string) -> { temp_c: number, condition: string }

For “what’s the weather in Tokyo,” the expected call has this function name and these argument values:

{ "name": "get_weather", "args": { "location": "Tokyo" } }

Your caller needs a valid call with the right tool name and arguments. JSON whitespace and object-key order need not match this example. A different function name, an invalid argument, or extra prose outside the JSON can still break the contract.

Measure reliability on the tools and requests you intend to support. Separate syntax failures from incorrect tool choices, bad argument values, and unnecessary calls; one aggregate success rate hides useful information.

{"name":"search","arguments":{"query":"weather in Oslo"}}
Valid JSONPassThe call can be parsed.
Expected toolPassThis task needs search.
Required argumentPassThe query field is present.
Argument typePassThe query is a string.

These are constructed examples, not model benchmark results. Passing these checks does not establish that a call is useful or authorized.

A constructed call passes or fails four checks: JSON parsing, tool-name lookup, required arguments, and argument types. These are validation examples, not measured model success rates.

Passing these four checks is necessary but does not establish that the call answers the user’s request. A schema-valid weather lookup for the wrong city is still wrong.

What "training it" actually means

This post uses SFT for tool-call training. The loss and optimizer are the same as in the SFT recipe; the examples now include calls and tool results.

A plain SFT example is a (prompt, response) pair:

prompt: "Write a haiku about the sea."
response: "Silent tide breathes in..."

A tool-use SFT example can contain a full trajectory: the request, an assistant call, a tool result, and an assistant answer. The tokenizer’s chat template serializes these messages. In the assistant-only recipe here, assistant targets receive loss and user/tool targets are masked. Gradients can still flow through the representations of masked input tokens.

Toy tool trajectory · assistant-only target loss

systemtarget loss masked
<|sys|>Youhavetools.
usertarget loss masked
<|usr|>WeatherinTokyo?
assistant · tool calltarget loss included
<|ast|><|call|>{"name":"weather","args":{"loc":"Tokyo"}}<|/call|>
tool · resulttarget loss masked
<|tool|>{"temp":18,"sky":"cloudy"}
assistant · answertarget loss included
<|ast|>18C,cloudy.<|end|>

Word chunks and delimiters are illustrative, not a real tokenizer or provider format. Masking removes direct target losses; assistant losses still backpropagate through computations that used the prompt and tool results.

Static role rows show a toy serialization. Assistant call and answer targets receive loss; other targets are masked. The displayed chunks are not an actual tokenizer’s token boundaries.

The call’s punctuation, field names, and argument values are part of the assistant output, as is its answer after reading the result. The real template determines which delimiters and end-of-turn markers also receive loss.

SFT increases the probability of the demonstrated calls. That can teach useful formatting and tool-selection patterns, but it does not guarantee valid JSON. A schema-aware decoder and a validator can still be useful at inference.

Minimal math

For this assistant-only recipe, use the SFT loss. If the training sequence is t1,t2,…,tnt_1, t_2, \dots, t_n and M⊆{1,…,n}M \subseteq \{1, \dots, n\} is the set of positions on assistant tokens (including the tool-call tokens), then the loss is:

L=−1∣M∣∑i∈Mlog⁡pθ(ti∣t<i)\mathcal{L} = -\frac{1}{|M|} \sum_{i \in M} \log p_\theta(t_i \mid t_{<i})

Tool results, the environment's reply, are also in the training sequence, because the model needs to condition on them to write the final answer. But they are not in MM. You don't want the model's gradients to memorize tool outputs as if it had to generate them itself; you want the model to treat them as input it reads. Masking the tool-result tokens out of the loss is what draws that line.

A worked trajectory

The opening example contains four messages. Here is how they become a training sequence.

The user sends: "What's the weather in Tokyo?"

The full training example, in logical form:

system:     "You have access to a get_weather tool."
user:       "What's the weather in Tokyo?"
assistant:  <tool_call>{"name":"get_weather","args":{"location":"Tokyo"}}</tool_call>
tool:       {"temp_c": 18, "condition": "cloudy"}
assistant:  "It's 18 C and cloudy in Tokyo right now."

A real chat template determines the serialized role headers and termination markers. Llama 3-family formats distinguish message boundaries, final answers, and tool handoffs; a generic <tool_call> block is only schematic. Use the exact tokenizer and template for the checkpoint rather than assembling a plausible-looking string.

In this assistant-only recipe, the call’s JSON and the final answer supply target losses; system, user, and tool-result targets are masked. Whether assistant headers and particular delimiters are scored depends on the template and loader.

A model may generalize from these examples to unfamiliar tools described in context. Test that separately from performance on tools included in training; the number of examples required depends on the base model, tool schemas, and tasks.

Where the data comes from

Synthetic data is one source of trajectories, alongside authored examples and appropriately collected execution logs. The papers use different recipes: ToolAlpaca simulates tool interactions, while Toolformer samples candidate API calls, executes them, and filters by their effect on continuation loss. Toolformer is not simply a stronger teacher generating full chat trajectories.

A constructed tool-training trajectory

Catalog
get_weather(location: string) → temperature_c: number
User
Is it cold in Paris?
Assistant call
get_weather({"location":"Paris"})
Tool result
{"temperature_c":7} — illustrative response, not live weather
Assistant answer
The tool reports 7°C in Paris.
Validate and retain
Check the call against the schema and the answer against the result before adding the example to the dataset.

Use real execution or a documented simulator. A teacher-written tool result alone does not validate the call.

A constructed weather trajectory shows generation, execution, validation, and acceptance. The displayed weather is illustrative; a teacher-written result alone does not establish correctness.

Evaluate the teacher on the task. Inspect its tool selection, arguments, and use of results. A stronger general model is not automatically a reliable source of trajectories for a particular interface.

Execute or document the simulator. ToolLLM constructs trajectories with a catalog of real APIs. Whether results come from execution or simulation, check that the answer follows from them.

Validate before accepting. Parse the call, check its schema, inspect execution failures, and check that the final answer uses the returned result correctly.

Check coverage as well as volume. Include ambiguous requests, missing arguments, unavailable tools, execution failures, and requests that need no call.

Keep a held-out set of requests and tool schemas so the training run can be assessed on more than formatting.

Parallel tool calls

Some requests contain independent calls. Looking up the weather in three cities can run concurrently if no lookup depends on another’s result. By contrast, searching for an account and then fetching its records requires a dependency between calls.

A supported interface can let one assistant turn request several independent calls. Training examples can demonstrate that pattern. The runtime dispatches them concurrently and returns each result with its matching call ID.

Sequential vs parallel tool calls · same three lookups
assumed timings

With these assumed durations: sequential 740 ms, concurrent 440 ms. Saving: 300 ms. Scroll across the timeline if needed.

0%25%50%75%100%fraction of sequential elapsed time →SEQUENTIALplanTokyowaitPariswaitNYCwaitanswerPARALLELplan3 callsrun togetheranswerin this example: ~41% sooner
Sequential: complete. Parallel: complete.
planemit tool calltool runsfinal answer

Sequential and concurrent execution of three independent lookups under assumed timings. Actual latency depends on call durations, orchestration, and model generation time.

Include parallel-call examples when that behavior is part of the target interface. The model also needs examples of dependencies that require sequential calls.

Implementation sketch

Here is one logical training record, pretty-printed for readability. It uses an OpenAI-style message shape, not Llama’s native serialized token format. The custom loss_mask field requires loader support; it is not a standard training API field. Actual JSONL stores each record on one line.

{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant. You have access to: get_weather(location: string) -> { temp_c: number, condition: string }."
    },
    {
      "role": "user",
      "content": "What's the weather in Tokyo?"
    },
    {
      "role": "assistant",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {
            "name": "get_weather",
            "arguments": "{\"location\": \"Tokyo\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": "{\"temp_c\": 18, \"condition\": \"cloudy\"}"
    },
    {
      "role": "assistant",
      "content": "It's 18 C and cloudy in Tokyo right now."
    }
  ],
  "loss_mask": {
    "system": false,
    "user": false,
    "tool": false,
    "assistant": true
  }
}

A custom training loader applies the checkpoint’s chat template, locates assistant target spans, and builds labels with other targets ignored. Correct role boundaries and masking are part of the implementation; once those are established, the SFT loss applies.

In this OpenAI-style record, function.arguments is an escaped JSON string. Other loaders, including some Hugging Face chat-template interfaces, expect an argument object. Convert to the format your tokenizer and training loader actually accept.

Misconceptions

"Constrained decoding makes fine-tuning unnecessary." A correct grammar-constrained decoder can restrict completed outputs to valid syntax, subject to the supported grammar and termination handling. A truncated response may still be incomplete. Syntax constraints do not establish the correct tool, argument values, or need for a call. Prompting, training, decoding, and runtime validation address different parts of that problem; see Structured Generation.

"Fine-tuning teaches only the tools seen during training." A model can learn to read a new schema in context, but generalization needs evaluation. Conversely, it may invent a tool that was not supplied. Restrict dispatch to the application’s allowlist regardless of what the model emits.

"There is a fixed minimum dataset size." Data requirements depend on the starting model and the behavior being added. Use learning curves and held-out execution tests to decide whether more examples improve performance.

What's next

The next post covers long-context training techniques, where the problem shifts from "what does the model emit" to "how does the model handle inputs that are 10x or 100x longer than anything it saw in pretraining."

Additional reading (and watching)