# Supervised Fine-Tuning

Training on curated demonstrations of desired behaviour — the first and most important stage of post-training.

**Rests on:** Instruction Tuning, Pre-Trained Model Adaptation

*LLMs and Generative AI · Post-Training Alignment*


## Why learn this

You have likely interacted with chatbots that feel disjointed or overly verbose before they were polished for public use. Supervised fine-tuning is the process of taking a raw, general-purpose text generator and teaching it to act like a helpful assistant who follows instructions clearly. Without this step, the model remains a library book that knows everything but refuses to summarize. This stage matters because it establishes the basic format of communication: if you ask for a list, you get a list; if you ask for a summary, you get a short paragraph. It is the essential bridge between knowing facts and being able to use them in a conversation. Without this foundation, later attempts to make the model 'polite' or 'correct


## The intuition

Imagine a student who has read every book in the world but has never spoken in a classroom. If you ask, "What is the capital of France?" they might launch into a three-hour lecture on French history because that is how information flows in the books they memorized. Supervised fine-tuning is like giving this student thousands of flashcards. Each card shows a question on one side and the perfect, concise answer on the other. For example: Prompt: "Summarize this email." Response: "The client wants to meet at 2 PM." The student does not learn new facts about emails; they learn the *pattern* of how to respond. They adjust their internal habits so that when they see a request for a summary, they automatically look for brevity rather than detail. You are not teaching them what to think; you are teaching them how to format their thoughts. The quality of these examples is critical. If you show them sloppy, rambling answers as the "correct" way to speak, they will learn to be sloppy and rambling. A thousand clear, well-structured examples will create a much better assistant than a million messy ones.


## An analogy

Supervised fine-tuning is like coaching an actor who has memorized every script in history but doesn’t know how to improvise on stage. The director (the trainer) provides hundreds of scenes with specific cues: "When asked for directions, point left and speak calmly." The actor practices these specific interactions until the response becomes automatic muscle memory. The actor already knows all the words (pretraining), but the coach is shaping *how* those words are delivered in real-time interaction. Note that this analogy has limits: unlike an actor, the model cannot ask "Why did you want it calm?" or understand the emotional context; it is merely mimicking statistical patterns based on the examples shown.


## Definition

Supervised fine-tuning is the process of training a language model on a set of paired examples where each input (prompt) is matched with a specific, desired output (response), teaching the model to mimic that exact style and format for future requests.


## How it connects

This sits between pretraining, which gives the model basic language knowledge, and preference optimization, which refines the model’s choices between good and bad answers. It also relates to task-specific adaptation, where a general model is specialized for a particular job like translation or coding.


## The mathematics

Supervised Fine-Tuning (SFT) is a way of teaching a pre-trained language model to behave like a helpful assistant. The pre-trained model already knows how to predict the next word in a sentence based on general internet text. SFT changes its internal settings so that it prefers the style and content of specific, high-quality examples.

The core mathematical idea is probability. When the model generates text, it does not just pick one word; it assigns a likelihood (a number between 0 and 1) to every possible next word. A higher number means the model thinks that word is more likely to be correct.

To teach the model, we give it pairs of examples: a prompt (the question or start of the sentence) and a desired response (the good answer). We want the model to assign a very high probability to the words in the desired response. To measure how well it is doing, we calculate a single number called the 'loss'.

Here is how the loss is calculated for one example:

1. Look at the desired response word by word.
2. For each word in the response, check what probability the model assigned to that specific word, given the prompt and all the words before it in the response.
3. Take the natural logarithm of that probability. The natural logarithm is a mathematical function that turns small probabilities into negative numbers. For example, the log of 0.5 is about -0.69. The log of 1.0 is 0.
4. Sum these logarithmic values for all words in the response.
5. Take the negative of that sum. This flips the sign so that a 'good' prediction (high probability, log near 0) results in a small positive loss, and a 'bad' prediction (low probability, large negative log) results in a large positive loss.

The formula for this process is:

Loss = - (log(probability of word 1) + log(probability of word 2) + ... + log(probability of word n))

In this formula:
- 'Loss' is the error score. Lower is better.
- 'log' is the natural logarithm function.
- 'probability of word i' is the chance the model gave to the i-th word in the correct answer.
- The minus sign ensures that high probabilities (which have logs close to zero) result in low loss values.

The training process then adjusts the model's internal numbers (parameters) to make this Loss value smaller. It does this by calculating how much each parameter contributed to the current error and nudging them in the direction that reduces the error. This adjustment uses calculus, specifically derivatives, but you do not need to perform these calculations yourself. The key takeaway is that SFT works by maximizing the probability of the correct answers in your dataset.


## A worked example

Let us walk through the calculation of the loss for a single word choice.

Scenario:
- Prompt: 'The sky is'
- Desired response word: 'blue'
- The model currently assigns these probabilities to possible next words:
  - P('blue') = 0.3 (30% chance)
  - P('gray') = 0.5 (50% chance)
  - P('red') = 0.2 (20% chance)

We want the model to prefer 'blue'. Currently, it prefers 'gray'. We calculate the loss for choosing 'blue' as the correct answer.

Step 1: Identify the probability of the correct word.
- The correct word is 'blue'.
- The model's probability for 'blue' is 0.3.

Step 2: Calculate the natural logarithm of this probability.
- We need log(0.3).
- Using a calculator, log(0.3) ≈ -1.204.
- Note: Because 0.3 is less than 1, its logarithm is negative.

Step 3: Apply the negative sign to find the loss contribution.
- Loss = - (log(0.3))
- Loss = - (-1.204)
- Loss = 1.204

Interpretation:
- A loss of 1.204 is moderate. It means the model was not very confident in the correct answer.
- If the model had been more confident, say P('blue') = 0.9:
  - log(0.9) ≈ -0.105
  - Loss = - (-0.105) = 0.105
  - This is a much smaller loss, which is better.
- If the model had been very wrong, say P('blue') = 0.01:
  - log(0.01) ≈ -4.605
  - Loss = - (-4.605) = 4.605
  - This is a large loss, indicating the model needs significant adjustment.

The training process will adjust the model's internal settings to increase P('blue') from 0.3 toward 0.9 or higher, thereby reducing the loss from 1.204 toward 0.


## From scratch, in Python

```python
# We'll simulate a tiny model that predicts the next word.
# Our model has three possible words: 'a', 'b', 'c'.
# We'll represent its 'knowledge' as a tiny table of probabilities for each context.

# Our training example: prompt 'start', desired next word 'a'.

# Step 1: the model's current probabilities for the next word after 'start'.
# These are like the weights we want to adjust.
probs = {'a': 0.3, 'b': 0.5, 'c': 0.2}

# Step 2: compute the loss (negative log of the probability of the correct word).
import math
correct_word = 'a'
p = probs[correct_word]
loss = -math.log(p)
print('Loss:', loss)  # about 1.204

# Step 3: pretend we've adjusted the probabilities (this is what training does)
# to make the correct word more likely.
new_probs = {'a': 0.7, 'b': 0.2, 'c': 0.1}
new_loss = -math.log(new_probs[correct_word])
print('New loss:', new_loss)  # about 0.357, lower is better

# This is the core of SFT: adjust probabilities to make the loss smaller.
```


## What the code does

The code defines a dictionary `probs` that maps each possible word to its probability. This is a lookup table, just like a real model's output layer (though a real one has millions of entries). The line `import math` brings in the logarithm function. Then we look up the probability of the correct word and compute `-math.log(p)`. The key idea is that the loss is small when the model is confident and correct. The next section shows a 'trained' version with a lower loss, illustrating what fine-tuning achieves. In real SFT, you'd use PyTorch and a function like `torch.nn.CrossEntropyLoss`, but the math is exactly this.


## Common misconceptions

**Fine-tuning teaches the model new facts** — If I fine-tune on a dataset about medical conditions, the model will know new medical facts it didn't know before.

SFT mainly adjusts the style and format of responses, not the underlying knowledge. The model's factual knowledge comes from pretraining on massive text corpora. If you fine-tune on a small dataset of medical Q&As, the model may learn to phrase medical answers well, but it hasn't learned new medical facts — it's just more likely to produce plausible-sounding phrases. In fact, fine-tuning on a small dataset can reinforce wrong answers if your demonstrations contain errors. In fact: SFT shapes the model's behaviour — how it structures an answer, how verbose to be, what tone to use — not its factual knowledge. The knowledge is already there from pretraining.

**More demonstrations always means a better model** — I should gather as many examples as possible for fine-tuning, even if some are low quality, because the model can learn from the good ones.

The model learns from all examples, including the bad ones. If your dataset has a bunch of poorly written or incorrect responses, the model will pick up those patterns. Your library notes explicitly say 'Demonstration quality dominates quantity.' A million messy examples will produce a messy model; a few hundred polished ones will produce a coherent, consistent style. In fact: Focus on curating a small set of high-quality demonstrations. The model is a copycat — it replicates whatever you show it, good or bad.

**SFT is the same as pretraining** — Supervised fine-tuning is just another round of pretraining with some new text.

Pretraining uses raw text and predicts the next word from any context; SFT uses prompt–response pairs, where the prompt is explicit and the response is a demonstration of desired behaviour. The loss is applied only to the response tokens, not to the prompt. It's a much more targeted adjustment. In fact: Pretraining gives the model general language ability. SFT adapts that ability to specific behaviour patterns by learning from demonstrations on a curated, task-specific dataset.

**Fine-tuning is always the best way to improve a model** — Whenever I want a model to do something new, I should fine-tune it on examples of that thing.

Fine-tuning is expensive and can hurt the model's general abilities (catastrophic forgetting). Your library cites 'The Craft of Post-Training' advising a 'steering test' to decide whether to fine-tune at all. Cheaper interventions like prompt engineering or retrieval-based augmentation might solve the problem with zero training. Fine-tuning is only worth it when the behaviour is hard to elicit through prompts or needs to be deeply internalised. In fact: Before fine-tuning, try cheaper methods: write a better prompt, or retrieve relevant information to include in the context. If that works, you've saved compute and avoided risks. Save fine-tuning for when you truly need the model to behave a certain way consistently.


## In short

Supervised fine-tuning is the first and most critical stage of post-training: you take a pretrained model and teach it how to behave by showing it curated demonstrations of desired responses. It learns format, style, and tone as much as content. The math is simple: adjust the model's probabilities so it gives more likelihood to the words in your good examples, and less to other words. The loss is the negative log-likelihood of the correct response. High-quality demonstrations matter more than quantity. Before fine-tuning, consider cheaper options like prompting or retrieval. In practice, SFT is the difference between a raw text predictor and a helpful assistant, whether on a server, a phone, or an agent loop.


## Where this is used

**Large Language Models.** In every LLM you’ve used — ChatGPT, Claude, Gemini — SFT is the first step after pretraining. The pretrained model is a text-completion engine; SFT on hundreds of thousands of instruction–response pairs turns it into an assistant. A developer at OpenAI would take the pretrained GPT-4 and fine-tune it on data like `{'prompt': 'Explain this concept in simple terms.', 'response': '[A well-written, concise explanation]}'. The model learns to follow instructions, to ask clarifying questions, and to apologise when wrong. The scale is enormous: often millions of examples, each one hand-written or generated and then filtered by human contractors. The result is that when you type a prompt, the model generates the beginning of a response that looks like the examples it was shown — not a continuation of random text on the internet.

**Small Language Models (on-device).** When you need a model to run on a phone or a laptop, you can’t use a 70-billion-parameter model. You use a smaller model, say 7 billion parameters, which has much less innate capability. SFT is even more critical here: you must carefully curate demonstrations that teach the model the specific tasks it will be used for — for example, a smart assistant on a phone that can set alarms, answer simple questions, and control smart home devices. The demonstrations are often narrowly focused, so the model becomes a specialist. The craft is in making the best possible use of the limited capacity: every demonstration must be high-value, teaching both content and style, because the small model can’t hold as much knowledge or pattern as a large one.

**Retrieval-Augmented Generation (RAG).** RAG systems retrieve relevant documents and stuff them into the context to ground the model’s answer. But the model needs to know how to use that retrieved text: how to cite it, how to say ‘According to the source…’, how to admit when the source doesn’t contain the answer. That behaviour is often taught with SFT. You’d create demonstrations where the prompt includes a user question and a retrieved passage, and the response is a well-grounded answer that references the passage. SFT teaches the model the *mechanics* of RAG — the formatting and reasoning — so that at inference time, when you plug in the retrieval system, the model knows what to do with it. Separately, SFT is used to train the models that generate the embeddings for retrieval, though that’s a different kind of fine-tuning.

**Agentic Systems.** When a model is used as an agent — planning, calling tools, and taking actions in a loop — SFT is essential to teach it the *format* of tool calls. A pretrained model has no idea that ‘{‘tool’: ‘get_weather’, ‘args’: ‘Bengaluru’}’ is a valid action. So you collect demonstrations of an agent successfully solving tasks: a prompt like ‘What’s the weather today in Bengaluru?’, followed by the agent’s reasoning, then a tool call, then the tool result, then the final answer. SFT teaches the model to emit that structured sequence: when to say ‘I should look that up’, what arguments to pass, and how to interpret the result. Without SFT, even a powerful pretrained model often fails to produce valid tool calls. This stage is a prerequisite for the agent to function even before you get to reinforcement learning on task success.

**Everyday Applied ML (Vision, Recommendation, etc.).** The same principle applies everywhere. In image classification, you take a model pretrained on ImageNet and fine-tune it on a few thousand medical X-rays with labels — that’s SFT. In recommender systems, you fine-tune a general embedding model on your user’s click history to teach it what ‘relevant’ means for your catalogue. The underlying mathematics is the same: adjust the model to produce the correct label or ranking for your curated examples. The only difference is the data type — in language it’s text; in vision it’s pixels; in a recommender it’s user and item features — but the loss function is the same negative log-likelihood on the correct answer.

**What a developer writes.** In practice, a developer uses a framework like Hugging Face Transformers. They load a pretrained model with `from_pretrained`, prepare a dataset of prompt–response pairs, and call `trainer.train()`. The job is mostly *data*: cleaning the data, filtering out bad examples, making sure the prompting format is consistent. The actual training is a few lines of code. The effort is in curation. The cost is large: training a 7-billion-parameter model on a single GPU can take days; training a 70-billion-parameter model takes hundreds of GPUs for weeks. The two things that break most often are overfitting on the small dataset (learning to memorise the demonstrations but not generalise to new prompts) and catastrophic forgetting (remembering the pretraining style less and less, so the model becomes worse at open-ended tasks). The craft of SFT — and your library says this — is deciding whether to fine-tune at all, and if so, curating demonstrations of the highest possible quality, because the model will amplify your dataset’s flaws.


---

Read this on the web: https://basavapurushottam.com/learn-ai/supervised-fine-tuning
