Special Tokens, Chat Templates, and Input Formatting

There's a layer between what you type and what the model sees, and it's easy to forget it's there. When you send a message to an LLM, your text gets wrapped in a precise formatting structure before it ever reaches the tokenizer. Special control tokens get injected. Role boundaries get marked. The whole multi-turn conversation history gets flattened into one long sequence of token IDs.

I think of this as the hidden formatting layer. It's invisible in every chat UI, and the model was trained against its exact output, which means it shapes behavior as concretely as the weights themselves. Use the wrong template with a fine-tuned model and you get quietly worse outputs, because the model is seeing token patterns it was never trained on.

In the BPE post we built a tokenizer from scratch and watched raw text become integer IDs. In the embedding-table post we saw how each of those IDs selects a row from the embedding table. Now we need to understand what happens before that tokenization step, and why the formatting choices matter so much.

Prerequisites: the tokenization picture from the four prior posts in this arc. Familiarity with JSON-ish message structures helps but isn't required.

The special token zoo

Every tokenizer ships with a set of special tokens that never appear in natural text. They exist purely to give the model structural information. The core set:

• BOS (Begin of Sequence). Marks the very start of a new input. The model has learned that this token means "fresh context, no prior state." Llama uses <s>, GPT-2 doesn't use one at all.
• EOS (End of Sequence). Signals that generation should stop. During training, this token teaches the model where sequences end. During inference, emitting EOS tells the decoding loop to halt.
• PAD (Padding). A filler token used to make sequences the same length in a batch. Every sequence in a training batch needs to be the same length for the GPU to process them in parallel. Shorter sequences get PAD tokens appended. The critical thing about PAD is that it must be masked out of both the attention computation and the loss function. If you let the model attend to padding or try to predict padding tokens, you are injecting noise into the gradients.
• UNK (Unknown). A fallback for tokens not in the vocabulary. Byte-level BPE (used by GPT-2, Llama, and most modern models) essentially eliminated UNK by encoding everything as byte sequences. Older tokenizers like BERT's WordPiece still have it.
• SEP (Separator). Used by BERT-family models to mark the boundary between two input segments, like a question and a passage in QA. Not used by decoder-only models.
• MASK. Specific to masked language models like BERT. A random subset of input tokens get replaced with [MASK], and the model learns to predict them. This is the core mechanism of MLM pre-training, and it doesn't exist in the autoregressive (GPT-style) world at all.

Here's what makes special tokens different from regular vocabulary. They get reserved token IDs in the tokenizer's vocabulary, and the tokenizer is explicitly told never to split them during tokenization. If you type the literal string <s> in a GPT prompt, the tokenizer will break it into separate subwords. When the system injects the BOS token, it maps directly to a single ID. The model's embedding table has a learned vector for that ID, trained on millions of examples where it appeared at the start of sequences.

In BERT-style models, the SEP and CLS tokens deserve a brief aside. BERT's input format for tasks like question answering looks like [CLS] question tokens [SEP] passage tokens [SEP]. The [CLS] token's final hidden state serves as the "whole-sequence representation" for classification tasks, and [SEP] tells the model where one segment ends and another begins. This design is specific to encoder models. Decoder-only models like GPT and Llama don't need SEP because they process text left-to-right and use different mechanisms to handle multiple segments.

Chat templates as model interface

Those are the classic special tokens. When chat-based fine-tuning became the norm in 2023, models needed a way to encode something new: conversational structure. Who said what, where one turn ends and another begins, and where the system instructions live.

The solution was to add more special tokens, purpose-built for chat. Each model family invented its own set.

Llama 3 uses a header-based system. Each turn starts with <|start_header_id|>, followed by the role name (system, user, or assistant), then <|end_header_id|>, then the content, then <|eot_id|> to close the turn. The whole sequence begins with <|begin_of_text|>.

ChatML, originally designed by OpenAI and now widely used by Qwen and other open models, takes a simpler approach. Each turn is wrapped in <|im_start|> and <|im_end|> (short for "instant message," the ChatML convention from OpenAI), with the role name right after the opening tag.

These are not cosmetic differences. Each of these tokens gets its own learned embedding, and the model was trained with them in exactly these positions. When the model sees <|start_header_id|>user<|end_header_id|>, it has learned through billions of examples during fine-tuning that what follows is a human request. That context influences every subsequent prediction.

One thing to be explicit about: the angle brackets in these token names are just a naming convention. The string <|start_header_id|> is a single atomic token with a single ID in the vocabulary. The tokenizer treats it as one indivisible unit, not five tokens (<, |, start, and so on). That's what keeps these special tokens genuinely never in natural text. No amount of prompt injection can synthesize them from user input, because the tokenizer's special-token handling is a separate pre-processing pass from normal subword splitting.

The thing to carry out of Arc 3: tokenization is a model interface, not a preprocessor step. The template, the special tokens, the role markers, together define the wire format the model was trained on.

Multi-turn formatting: flattening conversations

We interact with LLMs as conversations, back and forth, like texting. The model has no concept of "turns" or "messages" at any architectural level. It sees a single, flat sequence of token IDs, and it predicts the next one.

A three-turn conversation with a system prompt is really just a long document, formatted with special tokens to imply structure. The model has learned what those structural markers mean, but fundamentally it's doing the same thing it always does: predicting the next token given everything before it.

The system prompt typically sits at the very beginning of the sequence, right after BOS. It's always in the model's context window and always influences every subsequent token prediction through the causal attention mechanism. There's nothing special about system prompts at the architecture level. They're just text that comes first and uses a specific role marker.

Each turn boundary is marked by the template's end-of-turn token (<|eot_id|> in Llama 3, <|im_end|> in ChatML). The model has learned that this token means "this speaker is done, someone else is about to talk." When you see the model stop generating mid-response during inference, what happened is it emitted the end-of-turn token and the serving framework caught it.

Here's the same conversation formatted under two different templates. Toggle between them and hover over the token stream to see which conversation turn each token belongs to:

The conversation is identical in both cases. But the token sequences are meaningfully different. Different special tokens, different structural markers, different positions for role information. A model fine-tuned on Llama 3's format has learned to associate <|start_header_id|> with "a role label is coming next." Feed it ChatML's <|im_start|> instead, and those tokens are either out-of-vocabulary or map to completely unrelated embeddings.

Tool-call schemas as tokens

The visualization above includes a tool call, and that's where template formatting gets especially interesting. When a model needs to invoke an external tool (search an API, call a function, look something up), it has to somehow express a structured function call as a linear sequence of tokens.

Llama 3 handles this by introducing <|python_tag|> as a special token that signals "what follows is executable code." The tool response comes back under the role ipython, borrowing from the Python REPL metaphor. ChatML-derived formats often use XML-style delimiters like <tool_call> and </tool_call>. OpenAI's newer Harmony format goes further and puts the tool call on its own typed channel, separate from the regular assistant reply.

<|im_start|>assistant
<tool_call>
{"name": "get_weather",
 "arguments": {"location": "Tokyo"}}
</tool_call>
<|im_end|>

<|start|>assistant
<|channel|>commentary
to=functions.get_weather
<|constrain|>json
<|message|>
{"location": "Tokyo"}
<|call|>

There's a tension here. The model is a next-token predictor, trained to produce fluent natural language. Tool calls need it to output valid structured data, usually JSON or a function signature, where a single misplaced comma makes the output unparseable. The special tokens help by clearly delineating "this is structured output, not freeform text," giving the model a kind of mode switch. During fine-tuning, the model has seen thousands of examples where <|python_tag|> is followed by syntactically valid function calls, so it has learned to switch into a more structured generation mode when it sees that token.

Whatever the format, something on the serving side has to read it back out. Inference engines carry a tool-parser layer for exactly this (--tool-call-parser in vLLM), whose job is to spot the family-specific tool-call tokens in the model's output and lift them back up into a structured tool_calls field. It's the inverse of the template: tokens in, messages out. Part of what people mean when they say a model "supports tool use" is that this parser exists and works.

Some models go further and encode the tool definitions (function names, parameter schemas) directly into the system prompt as structured text. Before the conversation even starts, the model has seen a block of JSON or pseudo-code describing every tool available to it. The schema needs to appear in exactly the format the model was fine-tuned on, or the model's ability to correctly invoke tools degrades. We'll come back to tool use and agent loops in much more depth in Arc 10.

Jinja2 templates and apply_chat_template

So we have all these different formats. How do practitioners apply them?

Hugging Face solved this with a clever bit of standardization: every tokenizer now ships with a Jinja2 template stored in its tokenizer_config.json file. This template is a small program that takes a list of message dictionaries and produces the formatted string. You call it through the apply_chat_template method on the tokenizer.

Here's the anatomy of one. Watch the highlight cycle through the structural moves a chat template actually makes:

{{ bos_token }}
{% for message in messages %}
  <|start_header_id|>{{ message['role'] }}<|end_header_id|>\n\n{{ message['content'] | trim }}<|eot_id|>
{% endfor %}
{% if add_generation_prompt %}
  <|start_header_id|>assistant<|end_header_id|>\n\n
{% endif %}

from transformers import AutoTokenizer
 
tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct"
)
 
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user",   "content": "What is the capital of France?"},
]
 
# Apply the chat template stored with THIS tokenizer
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,            # return string, not token IDs
    add_generation_prompt=True  # append the assistant header
)
 
print(formatted)
# <|begin_of_text|><|start_header_id|>system<|end_header_id|>
#
# You are a helpful assistant.<|eot_id|>
# <|start_header_id|>user<|end_header_id|>
#
# What is the capital of France?<|eot_id|>
# <|start_header_id|>assistant<|end_header_id|>
#
 
# Now tokenize for real
token_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
)
print(f"Token count: {token_ids.shape[-1]}")
# Token count: 35

The key design decision here is that the template lives with the tokenizer, not with the model weights. If you download a model and its tokenizer from Hugging Face, the correct chat template comes bundled in. You don't need to look up the format documentation. You don't need to write formatting code. You call apply_chat_template and the right thing happens.

That bundling does more work than it looks like. vLLM and Text Generation Inference both expose an OpenAI-shaped /v1/chat/completions endpoint, and internally both of them call into the tokenizer's template to compile your messages array into tokens before anything reaches the GPU. Which makes the Jinja string the actual serving interface, not just a convenience for people calling transformers from a notebook. When you deploy Llama 3 on vLLM and POST a messages array at it, the template that shipped alongside the weights is what decides how your call becomes tokens.

This is also where things go wrong. If you fine-tune a model on data formatted with one template, then later load a tokenizer that has a different template (maybe from a different checkpoint, or a community upload with a modified config), the formatting silently changes. The model sees unfamiliar token patterns, outputs degrade, and no error is raised. This is what the Hugging Face team called "the silent performance killer" in their blog post announcing chat template standardization.

The Jinja template is also where tool definitions get injected. Templates that support function calling check for a tools parameter, and if present, render the tool schemas into the system prompt in the correct format. This is genuinely clever engineering. The same apply_chat_template call handles both plain conversation and tool-augmented conversation, and the model always sees exactly the format it was trained on.

Worked example

Let me make this concrete. A minimal three-turn conversation (system prompt, user message, assistant reply) rendered under four different template formats. Same messages, different token streams.

A few things jump out. Llama 3 uses more structural tokens than ChatML (separate start/end header markers, plus a begin-of-text token). Mistral's older format folds the system prompt into the first user turn. Harmony adds typed channels to the assistant turn. The overhead adds up across long conversations, and it's one of the practical considerations when choosing or designing a template format.

There's also a fork in the design space that API providers represent. Anthropic's Messages API accepts a structured array of messages and does the formatting server-side. Users never see the raw token stream. Open-source models mostly converge on "flatten everything into one token stream with special tokens," because the model literally needs a single sequence. API providers handle the structuring internally.

Misconceptions

"Chat format doesn't really matter." It does. During instruction fine-tuning, the model's loss function is computed over tokens in the assistant turns, conditioned on everything that came before them. If the structural tokens are wrong, the conditioning is wrong. The model has never seen this particular sequence of role markers during training, so its predictions are out-of-distribution.

"The system prompt is always just a separate role at the top." In ChatML and Llama 3, yes. In Mistral's default template the system prompt gets concatenated into the first user turn, because Mistral 7B v0.1 didn't have a dedicated system slot at all. Gemma doesn't have a system role either; instructions are expected to go in the first user turn. So an abstraction written around the assumption that there's always a system slot will quietly misplace instructions on a good chunk of the open-model ecosystem. Check the template.

"I can just write the special tokens as strings in my prompt." Not reliably. If you type <|im_start|> as literal characters, the tokenizer splits it into subwords: <, |, im, _start, |, > or something similar. Those subword IDs are not the same as the single special-token ID the model was trained on. You need the tokenizer's apply_chat_template method (or equivalent) to inject the real special tokens.

"PAD tokens are harmless filler." They're computationally harmless only if you mask them correctly. PAD tokens must be excluded from the attention mask (so other tokens don't attend to them) and from the loss computation (so the model isn't penalized for failing to predict padding). If your training loop gets this wrong, the model learns to waste capacity on predicting filler. We'll get into the details of attention masking and packing strategies in the next post.

"Special tokens are basically regex." They look like it, but they're resolved at the tokenizer level, not by string matching inside the content. That's a stronger guarantee: the tokenizer's pre-processing step refuses to decompose a registered special-token string into subwords, and also refuses to emit the special-token ID when it sees the matching string as raw user input. The two sides of this contract are what keep prompt injection from synthesizing its own <|eot_id|>.

What's next

That's the formatting layer. Special tokens give the model structural information, and chat templates flatten multi-turn conversations into the single token sequence the architecture actually consumes.

Next up is Packing, Masking, and Tokenization as a Model Interface, where we look at what happens to token sequences once they're batched together for training, and why the attention mask is what actually prevents cross-contamination between packed documents.

Additional reading (and watching)

• Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. Introduces [CLS] and [SEP] and the segment-based input format.
• Hugging Face. Chat Templates documentation. The canonical reference for apply_chat_template and the Jinja2 template format.
• Hugging Face Blog. Chat Templates: An End to the Silent Performance Killer. 2023. The announcement post that named the template-mismatch failure mode.
• Ronacher, A., et al. Jinja documentation. The template language every chat template is written in. Chat templates use the sandboxed subset with a couple of tokenizer-specific globals (bos_token, add_generation_prompt).
• vLLM Project. OpenAI Compatible Server. Documents how vLLM uses the tokenizer's shipped chat template to render incoming messages arrays, and how --chat-template and --tool-call-parser let you override the defaults.
• Hugging Face. Text Generation Inference: Messages API. TGI's OpenAI-compatible endpoint; same pattern as vLLM for invoking the model's native template server-side.
• OpenAI. (2023). ChatML format spec (archived). The original ChatML note that introduced the <|im_start|>…<|im_end|> delimiters later adopted by Qwen, Mistral-Nemo, and many others.
• Meta AI. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. Section on the Llama 3 instruction format, including <|start_header_id|>, <|end_header_id|>, <|eot_id|>, and <|python_tag|>.
• Jiang, A. Q., et al. (2023). Mistral 7B. arXiv:2310.06825. The paper that introduced the base Mistral [INST]…[/INST] template inherited from Llama 2.
• Touvron, H., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288. Section 3.2 specifies the original [INST], <<SYS>>, and system-prompt concatenation conventions.
• OpenAI. (2025). Harmony response format. Cookbook article introducing the <|start|>/<|channel|>/<|message|>/<|end|> transcript format used by gpt-oss and the o-series.