Special Tokens, Chat Templates, and Input Formatting

<|start_header_id|>user<|end_header_id|>

Explain entropy.<|eot_id|>

Before a chat model receives a conversation, a template serializes the messages into text and structural markers. The tokenizer converts that formatted conversation into one sequence of token IDs.

Most chat interfaces hide this formatting. A model is trained on particular conventions, so a mismatched template can change its behavior even when the visible messages are the same.

BPE maps text to IDs, and the embedding table maps those IDs to vectors. Chat templates specify the sequence to tokenize: who spoke, where messages end, and where the assistant should continue.

Prerequisites: the tokenization picture from the four prior posts in this arc. Familiarity with JSON-ish message structures helps but isn't required.

The special token zoo

Tokenizers reserve special token IDs for structural uses. The set varies by model; common examples include:

• BOS (Begin of Sequence). Marks a sequence start in models trained to use it. It does not reset an attention cache or impose an attention boundary. Llama 2 uses <s>; Llama 3 uses <|begin_of_text|>.
• EOS (End of Sequence). Marks a sequence end; serving code can use its ID as a stopping condition.
• PAD (Padding). Fills unused positions in rectangular batches. Mask its prediction targets and prevent real tokens from attending to padded keys. Variable-length implementations can avoid padding.
• UNK (Unknown). Represents text the vocabulary cannot encode. A complete byte alphabet or byte fallback can avoid unknowns for valid text; BERT's WordPiece tokenizer retains an unknown token.
• SEP (Separator). BERT uses it at segment boundaries.
• MASK. BERT's masked-language-model objective replaces some selected inputs with [MASK] and predicts the original tokens.

A registered special token can map to one reserved ID instead of being split into ordinary subwords. Whether a literal spelling in user text is recognized, rejected, or encoded as ordinary text depends on the tokenizer and its options. The template and tokenizer configuration have to agree.

BERT commonly formats paired inputs as [CLS] question [SEP] passage [SEP]. Its classification head can use the final hidden state at [CLS].

Chat templates as model interface

Model families use different combinations of special tokens and ordinary text for these boundaries.

Llama 3 uses a header-based system. Each turn starts with <|start_header_id|>, followed by the role name (system, user, or assistant), then <|end_header_id|>, then the content, then <|eot_id|> to close the turn. The whole sequence begins with <|begin_of_text|>.

ChatML, originally designed by OpenAI and now widely used by Qwen and other open models, takes a simpler approach. Each turn is wrapped in <|im_start|> and <|im_end|> (short for "instant message," the ChatML convention from OpenAI), with the role name right after the opening tag.

The model learns to use these markers during training. In Llama 3's format, <|start_header_id|>user<|end_header_id|> introduces a user message. That role and its content become part of the context for subsequent predictions.

Special tokens are not a prompt-injection defense: applications must handle untrusted content correctly, and a model can follow malicious instructions even without a forged role delimiter.

Preprocessing is part of the model interface, so its artifacts need versioning alongside the weights.

Multi-turn formatting: flattening conversations

A conversation can be serialized into one document, with markers identifying its messages and roles. The model conditions its next-token predictions on that formatted history.

A system message often appears near the beginning, though templates differ. When retained in context, its tokens are available through causal attention. Truncation can remove those instructions, so the retained history needs to preserve any system message the model is meant to use.

End-of-turn tokens mark message boundaries. A serving runtime can stop when the assistant emits the appropriate marker. A response can also end because of a length limit, configured stop string, cancellation, or an error, so a truncated response does not prove an end token was generated.

Here's the same conversation formatted under two different templates. Toggle between them and tap or hover over the stream to see which conversation turn each token belongs to:

The two templates encode the same messages with different markers and role positions. A checkpoint trained with Llama 3's header tokens has not necessarily learned to interpret ChatML's markers; the alternate spellings may also split into ordinary tokens.

Tool-call schemas as tokens

A tool call adds structured output to the conversation: a function name and arguments encoded as tokens for the serving runtime to parse.

Tool formats are model-specific. Llama 3.1 supports built-in tool conventions using ipython and <|python_tag|>, as well as function-call formats. ChatML-derived models can place structured calls inside <tool_call> delimiters. Harmony separates the recipient (such as to=functions.get_weather) from the channel (such as commentary), and uses a handoff marker to yield control.

<|im_start|>assistant
<tool_call>
{"name": "get_weather",
 "arguments": {"location": "Tokyo"}}
</tool_call>
<|im_end|>

<|start|>assistant to=functions.get_weather
<|channel|>commentary
<|message|>
{"location": "Tokyo"}
<|call|>

The runtime must parse the generated function name and validate its arguments against the tool schema.

Whatever the format, something on the serving side has to read it back out. Inference engines carry a tool-parser layer for exactly this (--tool-call-parser in vLLM), whose job is to spot the family-specific tool-call tokens in the model's output and lift them back up into a structured tool_calls field. It's the inverse of the template: tokens in, messages out.

Some models go further and encode the tool definitions (function names, parameter schemas) directly into the system prompt as structured text. Before the conversation even starts, the model has seen a block of JSON or pseudo-code describing every tool available to it. Use the checkpoint's supported tool-definition format and validate its behavior on your calls. We'll come back to tool use and agent loops in much more depth in Arc 10.

Jinja2 templates and apply_chat_template

Hugging Face Transformers supports Jinja templates associated with a tokenizer. Many instruction-tuned checkpoints supply one; base checkpoints may not. apply_chat_template renders the message dictionaries using that template. Use the template shipped for the checkpoint, and inspect it if the roles or tool schema do not match your application.

{{ bos_token }}
{% for message in messages %}
  <|start_header_id|>{{ message['role'] }}<|end_header_id|>\n\n{{ message['content'] | trim }}<|eot_id|>
{% endfor %}
{% if add_generation_prompt %}
  <|start_header_id|>assistant<|end_header_id|>\n\n
{% endif %}

from transformers import AutoTokenizer
 
tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct"
)
 
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user",   "content": "What is the capital of France?"},
]
 
# Apply the chat template stored with THIS tokenizer
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,            # return string, not token IDs
    add_generation_prompt=True  # append the assistant header
)
 
print(formatted)
# Simplified structural excerpt, not exact output: the shipped Llama 3.1
# template also supplies metadata such as its knowledge-cutoff/date fields.
# <|begin_of_text|><|start_header_id|>system<|end_header_id|>
#
# You are a helpful assistant.<|eot_id|>
# <|start_header_id|>user<|end_header_id|>
#
# What is the capital of France?<|eot_id|>
# <|start_header_id|>assistant<|end_header_id|>
#
 
# Now tokenize for real
token_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
)
print(f"Token count: {token_ids.shape[-1]}")
# The exact count depends on the checkpoint and rendered template.

vLLM and Text Generation Inference both expose an OpenAI-shaped /v1/chat/completions endpoint, and internally both of them call into the tokenizer's template to compile your messages array into tokens before anything reaches the GPU. The Jinja template therefore affects requests served through the API as well as local transformers calls. When you deploy Llama 3 on vLLM and POST a messages array at it, the template that shipped alongside the weights is what decides how your call becomes tokens.

Loading a different tokenizer template after fine-tuning can silently change the rendered sequence. The resulting predictions may degrade even though the request is syntactically valid. Inspecting the rendered text and token IDs can reveal that mismatch.

Templates that support tools can accept definitions through a tools argument and render them in the expected format.

Worked example

Here are the same three messages (system prompt, user message, assistant reply) rendered under four template formats.

Llama 3 uses separate start/end header markers and a begin-of-text token. The older Mistral format folds system instructions into the first user turn. Harmony includes assistant channels; this example ends a generated final reply with <|return|>, while its documented persisted-history form uses <|end|>. Each format adds its own token overhead.

Hosted APIs can expose structured messages while keeping token serialization internal. Anthropic's Messages API, for example, accepts message objects. Open-model deployments expose the tokenizer and template, letting you inspect the resulting sequence directly.

Misconceptions

"Chat format doesn't really matter." The format is part of the context on which the model was trained. In response-only instruction tuning, assistant targets are predicted from that context; other recipes also train on prompt targets. Changing role markers or their positions can change the conditioning even if message text stays the same.

"The system prompt is always just a separate role at the top." ChatML and Llama 3 support system messages. Mistral 7B v0.1 instead places system instructions in the first user turn; early Gemma instruction templates also omit a separate system role. Inspect the checkpoint's supported roles before rendering messages.

"A special-token spelling always becomes ordinary text." Some tokenizers recognize it as the reserved ID; others reject or split it under particular settings. Inspect the produced IDs and use the checkpoint's chat template to construct complete conversations.

"PAD tokens are harmless filler." They're computationally harmless only if you mask them correctly. PAD tokens must be excluded from the attention mask (so other tokens don't attend to them) and from the loss computation (so the model isn't penalized for failing to predict padding). If your training loop gets this wrong, the model learns to waste capacity on predicting filler. We'll get into the details of attention masking and packing strategies in the next post.

"Special tokens prevent prompt injection." They encode structure, not trust. A tokenizer can recognize a reserved spelling without knowing whether it came from a trusted template or an untrusted message. Delimiter handling and model behavior require separate attention.

What's next

Next up is Packing, Masking, and Tokenization as a Model Interface, where we look at what happens to token sequences once they're batched together for training, and why the attention mask is what actually prevents cross-contamination between packed documents.

Additional reading (and watching)

• Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. Introduces [CLS] and [SEP] and the segment-based input format.
• Hugging Face. Chat Templates documentation. The canonical reference for apply_chat_template and the Jinja2 template format.
• Hugging Face Blog. Chat Templates: An End to the Silent Performance Killer. 2023. The announcement post that named the template-mismatch failure mode.
• Ronacher, A., et al. Jinja documentation. The template language used by Transformers chat templates. Chat templates use the sandboxed subset with a couple of tokenizer-specific globals (bos_token, add_generation_prompt).
• vLLM Project. OpenAI Compatible Server. Documents how vLLM uses the tokenizer's shipped chat template to render incoming messages arrays, and how --chat-template and --tool-call-parser let you override the defaults.
• Hugging Face. Text Generation Inference: Messages API. TGI's OpenAI-compatible endpoint; same pattern as vLLM for invoking the model's native template server-side.
• OpenAI. (2023). ChatML format spec (archived). The original ChatML note that introduced the <|im_start|>…<|im_end|> delimiters later adopted by Qwen, other model families.
• Meta AI. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. Section on the Llama 3 instruction format, including <|start_header_id|>, <|end_header_id|>, <|eot_id|>, and <|python_tag|>.
• Jiang, A. Q., et al. (2023). Mistral 7B. arXiv:2310.06825. The paper that introduced the base Mistral [INST]…[/INST] template inherited from Llama 2.
• Touvron, H., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288. Section 3.2 specifies the original [INST], <<SYS>>, and system-prompt concatenation conventions.
• OpenAI. (2025). Harmony response format. Cookbook article introducing the <|start|>/<|channel|>/<|message|>/<|end|> transcript format documented for gpt-oss.