# Chat Format Templates

The role-tagged structure — system, user, assistant — that conversational models were trained to expect.

**Rests on:** Instruction Tuning, Token IDs and Special Tokens

**Leads to:** Conversational Agents, LLM as Study Partner

*LLMs and Generative AI · Fine-Tuning*


## Why learn this

Before you ever fine-tune a conversational model, you must shape your data into the exact role-tagged structure the model was trained on. Get this wrong and the model will look broken — rambling, ignoring instructions, refusing to answer — when the real fault is a mismatched template. This concept is the gate to everything in Fine-Tuning: Instruction Tuning, LoRA and QLoRA, DPO, and every one of your neighbouring notes that assumes data already sits in the right chat format. Master the template first and every later step — from a simple fine-tune to a multi-tenant adapter service — will behave the way the books say it should.


## The intuition

Modern chatbots are not programmed to hold conversations. They are trained on millions of transcripts that all follow the same invisible skeleton: each line is tagged with a role. There is a system role that sets the ground rules, a user role that speaks the actual request, and an assistant role that gives the answer. The model does not learn what a 'user' is from a dictionary — it learns by seeing the pattern of who talks when, and what the correct response looks like after each kind of prompt. Once training is complete, the same skeleton is used at inference time: you hand the model a text that begins with <|system|>, then <|user|>, and the model continues with the <|assistant|> line because that is the only continuation it has ever seen work.


## An analogy

Think of a model as a drama school student who has learned every play ever written, but the plays are all written in the same rigid format: a cast list at the top, then lines labelled 'King:', 'Queen:', 'Fool:'. If you hand this student a script that omits the cast list and the labels, they will flounder — not because they lack acting ability, but because they have never seen a script in that shape. The chat format template is the script format. For one model it might be 'You: ... Assistant: ...', for another '<|user|> ... <|assistant|>'. The rules of the script are exactly the rules the drama student memorised during training. If you give them a different formatting convention — say, a script with no labels at all — they cannot call up what they learned, because their knowledge is welded to the format. The format is not a superficial wrapper; it is part of the language. Where the analogy stops holding: a drama student can improvise and adapt to an unlabelled script because they have social intuition. A language model has no such intuition. It only knows the patterns in its training data. If the input strays from the format, the model does not 'figure it out' — it produces something, often odd, because it is completing a pattern that was not trained.


## Definition

A chat format template is the exact sequence of role tags — typically system, user, and assistant — that a conversational model expects to see wrapped around every message, because that is the only structure it was trained on.


## How it connects

You have not yet studied any other concepts, so everything here is new. But this idea connects directly ahead to your first fine-tuning steps. In Instruction Tuning, you will take raw instruction-response pairs and re-serialise them into the chat template before training a single weight. In LoRA and QLoRA, the adapter learns on data that must already be in this format — otherwise the adapter is learning the wrong associations. Even your note on Multi-Tenant Adapter Serving presumes the template is handled correctly per request. If the template is wrong, the model's behaviour degrades in ways that look like a weak model, so debugging begins here. Your book 'Fine-Tuning with Python' is emphatic: data must be shaped into the model's chat format before tuning, or nothing downstream works.


## The mathematics

This concept uses no mathematics in the sense of numbers or formulas — it involves sequences of tokens (words or symbols) and the order in which they appear. The relevant branch is set theory and formal languages, but only at the level of 'a sequence is a list of items in a specific order'. What you need to grasp is that a template is a fixed string pattern. For example, the pattern '<|user|> {message} <|assistant|>' is a sequence of four elements: a constant tag, a slot for the message, another constant tag, and another slot. The model was trained on text that always had these tags in this order, so at inference time, you must reproduce the same order. There is no arithmetic — no addition or multiplication — but there is a logic of concatenation: take the system message, prepend the system tag, take the user message, prepend the user tag, and so on, then join them all into one long string. The final output is a single string that you pass to the model.

A chat format template is a sequence of roles and contents: S = T_s + C_s + T_u + C_u + T_a + ... where T_s, T_u, T_a are the system, user, and assistant tags, and C_s, C_u are the corresponding content strings. The model's output is the continuation of this string after the final assistant tag.


## A worked example

```
Suppose a model uses the template <|system|> ... <|user|> ... <|assistant|>. You have three messages: a system prompt 'You are a helpful assistant.', a user message 'What is the capital of France?', and the desired assistant answer 'Paris.'

Step 1: Start with the system tag: <|system|>   # The template begins with the system tag.
Step 2: Append the system message: <|system|> You are a helpful assistant.   # The system content is placed after its opening tag.
Step 3: Append a newline (often used to separate roles): <|system|> You are a helpful assistant.
<|user|>   # A line break helps readability, but the key is the user tag appears.
Step 4: Append the user message: ...
<|user|> What is the capital of France?   # The user's text comes after the user tag.
Step 5: Add a newline and the assistant tag: ...
<|assistant|>   # The assistant tag marks where the model should begin generating.
Step 6: The full prompt string is: <|system|>You are a helpful assistant.
<|user|>What is the capital of France?
<|assistant|>   # The model will continue from this point, generating the assistant response 'Paris.'.

Result: The final assembled string is the input to the model. The model's generation starts immediately after the assistant tag, and the output is the assistant's message.
```


## From scratch, in Python

```python
# --- Building blocks ---
# A list is an ordered collection of items. Example:
messages = ["system message", "user message", "assistant message"]
# A dictionary maps keys to values, like a phonebook.
message = {"role": "user", "content": "What is the capital of France?"}

# --- Simple template without format enforcement ---
def format_chat_simple(messages):
    # messages is a list of dictionaries, each with 'role' and 'content'
    result = ""
    for msg in messages:
        role = msg["role"]
        content = msg["content"]
        result += role + ": " + content + "\n"
    return result

# --- A model-specific template, e.g. for Llama 2 ---
def format_chat_llama2(messages):
    # '<s>' is the start-of-sequence token
    formatted = "<s>"
    for msg in messages:
        role = msg["role"]
        content = msg["content"]
        if role == "system":
            formatted += "[INST] " + content + " [/INST]"
        elif role == "user":
            formatted += "[INST] " + content + " [/INST]"
        elif role == "assistant":
            formatted += " " + content + " "
    formatted += "</s>"
    return formatted

# --- Example usage ---
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]
print(format_chat_simple(messages))
print(format_chat_llama2(messages))
```


## The same thing, with a framework

In practice, you would use a library like Hugging Face Transformers. The function `apply_chat_template` on a tokenizer does exactly this. For example:

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False)
print(prompt)
```

Under the hood, it runs the same loop as `format_chat_llama2` — it reads the model's `chat_template` definition and applies it to the list of dictionaries.


## What the code does

The code first defines a list `messages` containing three strings to illustrate the concept of a list. Then it defines a dictionary `message` with two keys, 'role' and 'content', to show how a single message is represented. The first function `format_chat_simple` takes a list of such dictionaries, loops over each message with a for loop, extracts the role and content using square-bracket indexing, and builds a string by concatenating role + ': ' + content with a newline. This produces a crude template. The second function `format_chat_llama2` implements a more realistic template: it starts with '<s>', then for each message, if the role is 'system' or 'user', it wraps the content in '[INST] ... [/INST]'; if the role is 'assistant', it simply appends the content. At the end it adds '</s>'. The difference matters: different models expect different tag tokens. Using the wrong tags will confuse the model because it was trained on the correct ones.


## Common misconceptions

**Any order of roles is fine as long as all messages are present** — The order of messages does not matter as long as I include the user and assistant turns.

The model was trained on a very specific order: system first, then alternating user and assistant. Reordering breaks the pattern the model learned, so it may respond incorrectly or with lower quality. In fact: The order is part of the template and must be followed exactly. Each role appears in a strict sequence, and the model's generation starts after the final assistant tag.

**The template is just a cosmetic convenience** — The template is just for readability; the model can understand a plain text prompt without role tags.

Role tags are not decorative — they are part of the training data. The model's weights encode the pattern with tags; without them, the pattern is broken. In fact: The chat template is the expected input structure. Omitting it degrades performance in unpredictable ways.

**All models use the same chat template** — I can use the same template for Llama and Mistral; they are both chat models.

Each model family (and sometimes each checkpoint) defines its own tag tokens (e.g., '<|user|>' versus '[INST]'). Using the wrong one disrupts the model. In fact: Always match the template to the exact model you are using. The model card or tokenizer config states the correct template.

**System message is optional and can be omitted** — I can skip the system message; the model will still follow instructions.

Many models rely on the system prompt to set behaviour. Skipping it removes important instructions, but more importantly, the model may still generate an assistant turn correctly because it was trained with system messages present. Omitting it is a deviation. In fact: Include the system message in the correct position, even if empty, to match the training distribution.


## In short

The chat format template is the role-tagged structure (system, user, assistant) that a model was trained on. It is not optional and not interchangeable between models. When you build or fine-tune a conversational model, you must serialise every message into this exact format. Failure to do so looks like a weak model, but is a formatting bug. Use `apply_chat_template` from Hugging Face Transformers to get it right automatically.


## Where this is used

**In large language models (LLMs):** Every contemporary chat-capable model — Llama, Mistral, GPT, Claude — is fine-tuned on a specific chat format. When you interact with ChatGPT, your browser sends the entire conversation history as a list of messages; the API then serialises them using the model's template before inference. If you swap a model without updating the template, the model's output quality plummets — it may ignore system instructions, repeat itself, or emit random text. Developers using the `transformers` library call `apply_chat_template` to guarantee the format matches the model card.

**In small language models on devices (TinyML):** On phones or microcontrollers, models like Phi-3-mini or TinyLlama are tiny, but still expect the same template. A keyword-spotter or on-device assistant must construct the prompt in the format the model was trained on, because there is no cloud to fix mistakes. The template is hard-coded in the app's source, and a mismatch can cause the assistant to respond to the wrong wake words or fail to respond at all.

**In retrieval-augmented generation (RAG):** When you retrieve documents and feed them to an LLM, you must decide where they go in the chat template. The system message often contains the retrieved context, instructing the model to answer based only on that. The user message contains the question. If you stuff the context into the user message instead, the model may treat it as instruction rather than evidence, leading to hallucinated claims. The correct placement is part of the system prompt, which is part of the template.

**In agentic systems that plan and call tools:** Agents loop — they take a user request, plan, call a function, get a result, and then compose the next user or assistant turn. Each step must be serialised into the correct roles. The function result often goes into a special 'tool' role, which the template defines. If the template has no tool role, you must map it to a user or assistant message, or the agent will not learn to stop calling the tool. Agents are particularly sensitive because a single malformed turn breaks the entire trajectory.

**In everyday applied work (recommendation, forecasting, fraud detection):** These tasks often use classification fine-tuning, where a model is adapted to predict a label. Even there, the input text must be wrapped in the chat template so the base model's representations stay aligned. If you skip the template, the model sees out-of-distribution text and performs worse, purely due to a formatting issue. In practice, developers copy the template from the model card and never deviate.

**What developers actually write:** With Hugging Face, it is two lines: `tokenizer.apply_chat_template(messages, tokenize=False)` and then `tokenizer(prompt, return_tensors='pt')` to convert to tokens. With the OpenAI API, the `messages` list is already the chat format, and the API does the serialisation internally — but if you use a non-chat model endpoint, you must format yourself. The common failure: a developer copies a template from another model and it silently breaks, producing poor outputs that are misdiagnosed as model weakness. The fix is always to check the model's `tokenizer_config.json`.


---

Read this on the web: https://basavapurushottam.com/learn-ai/chat-format-templates
