# Instruction Tuning

Fine-tuning on instruction-response pairs so a model follows directions rather than merely continuing text.

**Rests on:** Pre-Trained Model Adaptation, Tokenization

**Leads to:** Chat Format Templates, Continuous Learning Loops, Direct Preference Optimization, Prompt Injection, Supervised Fine-Tuning, Synthetic Data Generation, The Steering Test

*LLMs and Generative AI · Fine-Tuning*


## Why learn this

Instruction tuning turns a raw text-prediction engine into an assistant that actually follows directions. Without this step, a model will simply continue your sentence as if you were in the middle of writing a story, rather than answering your question or performing the task you requested. Understanding this concept is essential because it explains why modern AI systems can hold conversations and complete complex tasks, while earlier versions of the technology could only guess the next word in a paragraph. It is the specific change that bridges the gap between 'a machine that generates text' and 'a machine that does what you ask'.


## The intuition

Imagine a person who has read every book in the library but has never had a conversation. If you walk up to them and say, 'The weather is nice today,' they might start reciting a paragraph about meteorology or continuing a story about clouds, because that is what their training consisted of: predicting what comes next in a text stream. They do not know that your sentence was an attempt to start a dialogue. Now imagine teaching this person how to converse. You show them thousands of examples of exchanges. In each example, you present a clear question or command, followed by the correct response. For instance:

Input: 'What is the capital of France?'
Output: 'The capital of France is Paris.'

Input: 'Please summarize the following text...' [Text about climate change]
Output: 'This text explains that global temperatures are rising...'

After seeing enough of these pairs, the person stops just guessing the next word. Instead, they learn a pattern: when someone addresses them with a specific prompt format, they should respond directly to it. They learn that a question mark implies a request for information, and a command like 'Summarise' implies a need for condensation. They do not memorize every specific book or answer; they learn the general rule of engagement. This allows them to handle new questions they have never seen before, simply by recognizing the form of the instruction. The result is a partner who listens and responds appropriately, rather than a mimic who just keeps talking.


## An analogy

Think of a raw language model as a brilliant but rigid clerk who has memorized every document in an office building but has never been told how to help a visitor. If you ask, 'Where is the meeting room?' they might begin reading out the full table of contents of the office directory because their only job has been to reproduce text sequences they have seen before. To fix this, you do not send them back to read more books. Instead, you give them a guidebook filled with example interactions: 'Visitor says X, Clerk should say Y.' After reviewing hundreds of these examples, the clerk learns the new rule: when a visitor speaks, provide the specific answer or action requested, not just related text. This guidebook is the instruction dataset, and the learning process is instruction tuning. The analogy holds because both involve changing behavior through example rather than new information acquisition, but it breaks down in one way: the clerk might understand the context of 'meeting room' based on human intuition, whereas the model only recognizes the statistical likelihood that certain words follow others.


## Definition

Instruction tuning is the process of training a language model on paired examples of prompts and responses so that it learns to follow explicit directions and answer questions, rather than merely predicting the next word in a text sequence.


## How it connects

This concept follows directly from Large Language Models, where you learned how models are trained to predict text, and Pretraining, which provides the raw knowledge base. It serves as the foundation for Parameter-Efficient Fine-Tuning, which is a method for adjusting only a small part of the model's settings to save time and money while still teaching it to follow instructions. It also connects to Continued Pre-Training, which involves teaching the model new vocabulary before it learns to follow instructions about that new subject.


## The mathematics

Instruction tuning is a method for teaching a large language model to behave like a helpful assistant. It does not require new mathematics, but it relies on two familiar ideas: comparing numbers (to see how wrong the model is) and adjusting those numbers (to make the model better). 

The process works in three stages:

1. The model makes a guess.
2. We measure how far that guess is from the correct answer.
3. We change the model’s internal settings slightly to reduce that error.

Here is what happens in detail.

Stage 1: The Model Makes a Guess
A large language model predicts the next word in a sequence. It does this by assigning a probability to every possible word in its vocabulary. For example, if the input is "The sky is", the model might assign:
- "blue": 0.6 (60% chance)
- "gray": 0.3 (30% chance)
- "rain": 0.1 (10% chance)

In instruction tuning, the "input" is not just a sentence fragment. It is an instruction, such as "What is the capital of India?" The model must generate the correct response, "New Delhi." The mathematics of generating this response is the same as in ordinary text generation: the model calculates probabilities for each next word based on all previous words.

Stage 2: Measuring the Error (The Loss Function)
Once the model has produced a full response, we compare it to the correct answer. We use a number called the "loss" to quantify how bad the response was. The loss is always a positive number. A loss of zero means the model was perfect. A large loss means the model was very wrong.

The most common way to calculate this error for simple cases is by squaring the difference between the predicted value and the target value. This is called the "squared error." 

Formula: Loss = (prediction - target) squared

Let us look at a simplified example where we are predicting a single number instead of a whole sentence. Suppose the correct answer (the target) is 10. The model predicts 8.

Step A: Find the difference.
Difference = prediction - target = 8 - 10 = -2

Step B: Square the difference.
Loss = (-2) * (-2) = 4

If the model had predicted 9, the difference would be -1, and the loss would be 1. Since 1 is smaller than 4, we know the prediction of 9 was better than the prediction of 8. The goal of training is to find settings (called "weights") that make this loss as small as possible.

Stage 3: Adjusting the Weights (Gradient Descent)
The model has millions or billions of internal numbers called weights. These weights determine how the input text is transformed into the output text. To reduce the loss, we must change these weights.

We cannot try every possible combination of weights because there are too many. Instead, we use a method called gradient descent. This method tells us which direction to move each weight to lower the loss.

How do we know which direction? We look at the "slope" of the loss function. In calculus, this slope is called the derivative. For our squared error example, the slope tells us how much the loss changes if we change the weight by a tiny amount.

If the slope is negative, it means that increasing the weight will decrease the loss. If the slope is positive, it means that decreasing the weight will decrease the loss.

The update rule for one weight is:
New Weight = Old Weight - (learning rate * slope)

Let us apply this to our numerical example.

Suppose we are adjusting one specific weight, w. 
- Current weight (w) = 3.0
- Learning rate = 0.1 (this is a small number that controls how big each step is)
- Slope (derivative of loss with respect to w) = -2.0

Step A: Calculate the adjustment amount.
Adjustment = learning rate * slope = 0.1 * (-2.0) = -0.2

Step B: Subtract the adjustment from the old weight.
New Weight = 3.0 - (-0.2)
Note: Subtracting a negative number is the same as adding.
New Weight = 3.0 + 0.2 = 3.2

Why did we add? Because the slope was negative, meaning increasing the weight would reduce the loss. So we increased the weight by 0.2.

We repeat this process for every weight in the model, many times, over many examples. Each time, the weights shift slightly toward values that produce lower loss.

Instruction Tuning vs. Ordinary Training
The mathematics described above is identical to ordinary training. The only difference is the data used.
- In ordinary training, the model learns from plain text (novels, articles, etc.). It learns grammar and facts.
- In instruction tuning, the model learns from pairs of (instruction, response). For example:
  Input: "Summarize this article."
  Output: "The article argues that..."

By training on these pairs, the model learns not just what words come next, but how to follow instructions. The loss function still measures the difference between the generated output and the desired output. The weight updates still use gradient descent.

Efficiency Note: Low-Rank Adaptation (LoRA)
Training all billions of weights is expensive. A common technique called LoRA (Low-Rank Adaptation) reduces the number of weights we need to update. Instead of changing all weights, we add small "adapter" matrices to specific parts of the model. These adapters have far fewer parameters than the original model. The mathematics of updating these adapter weights is the same gradient descent described above, but applied to a much smaller set of numbers. This makes instruction tuning faster and cheaper while achieving similar results.


## A worked example

Let us walk through one step of training for a tiny model that predicts a single number. We will use the squared error loss function.

Setup:
- Target value (what we want): 10
- Model's current prediction: 8
- Current weight (w): 3.0
- Learning rate (lr): 0.1

Step 1: Calculate the Loss
The loss measures how far the prediction is from the target.
Formula: Loss = (prediction - target) squared
Calculation: (8 - 10) squared = (-2) squared = 4
Result: The loss is 4.

Step 2: Calculate the Slope (Derivative)
The slope tells us how to change the weight to reduce the loss. For squared error, the slope with respect to the prediction is 2 * (prediction - target). However, we need the slope with respect to the weight. In this simplified example, let us assume the prediction is directly equal to the weight for demonstration purposes (so prediction = w).
Slope = 2 * (w - target)
Calculation: 2 * (3.0 - 10) = 2 * (-7) = -14
Result: The slope is -14.
Note: A negative slope means that increasing the weight will increase the prediction, which will bring it closer to the target of 10 (since 8 < 10), thus reducing the loss.

Step 3: Update the Weight
Formula: New Weight = Old Weight - (learning rate * slope)
Calculation: 3.0 - (0.1 * -14)
First, multiply: 0.1 * -14 = -1.4
Then, subtract: 3.0 - (-1.4) = 3.0 + 1.4 = 4.4
Result: The new weight is 4.4.

Step 4: Verify the Improvement
Now that the weight is 4.4, let us see what happens to the loss if we use this new weight as the prediction (again, assuming prediction = w for this simple example).
New Prediction: 4.4
New Loss: (4.4 - 10) squared = (-5.6) squared = 31.36
Wait, the loss increased! This is because our simplified assumption (prediction = weight) was too crude for this large step. In real models, the relationship between weight and prediction is complex, and the learning rate is usually smaller to prevent overshooting.

Let us try a more realistic scenario where the prediction is closer to the target.

Revised Setup:
- Target: 10
- Current Prediction: 9.5
- Current Weight (w): 3.0
- Learning rate: 0.01 (smaller step)
- Assume prediction = w * 3.166... (so 3.0 * 3.166 ≈ 9.5)

Step 1: Loss
(9.5 - 10) squared = (-0.5) squared = 0.25

Step 2: Slope
Slope with respect to prediction = 2 * (9.5 - 10) = -1.0
Assume the slope with respect to weight is proportional, say -0.3.

Step 3: Update Weight
New Weight = 3.0 - (0.01 * -0.3)
= 3.0 - (-0.003)
= 3.003

Step 4: New Prediction
New Prediction ≈ 3.003 * 3.166 ≈ 9.51
New Loss = (9.51 - 10) squared = (-0.49) squared = 0.2401
Result: The loss decreased from 0.25 to 0.2401. The model is now slightly closer to the correct answer.

Conclusion:
By repeating this process thousands of times, the weights adjust until the loss is minimized, meaning the model’s predictions match the desired responses as closely as possible.


## From scratch, in Python

```python
# We will build a tiny instruction-tuning loop from scratch.
# This is a demonstration: in real life, models are huge and use libraries like PyTorch.
# But the core idea is the same: take a prediction, compute loss, update weights.

# First, let's create a very simple model: a single weight.
weight = 3.0  # starts at some value
learning_rate = 0.1

# Our training data: one instruction-response pair.
# For simplicity, we pretend the model outputs a single number (the probability of 'New Delhi' being the next word).
training_data = [
    ("What is the capital of India?", "New Delhi")
]

# We'll do one training step.
# In practice, we would tokenize the text and run it through the model.
# Here, we just make up a prediction for demonstration.
prediction = 0.3  # model's current output (should be close to 1.0 if correct)
target = 1.0      # we want the model to predict 'New Delhi' with high probability

# Compute the loss (mean squared error).
loss = (prediction - target) ** 2
print(f"Initial loss: {loss}")

# Compute the derivative of the loss with respect to the weight.
# For our made-up model, we pretend the derivative is 2 * (prediction - target) * (some factor).
derivative = 2 * (prediction - target)  # simplified

# Update the weight.
weight = weight - learning_rate * derivative
print(f"Updated weight: {weight}")

# Recompute the prediction and loss after the update.
# (In a real model, the prediction would change; we just show the loss decreasing.)
new_loss = (0.4 - target) ** 2  # suppose prediction improved to 0.4
print(f"New loss: {new_loss}")

# This loop would repeat over many examples and many iterations.
# The key is that the loss decreases, and the model gets better at following instructions.

```


## The same thing, with a framework

In practice, you would use PyTorch or another deep learning library. The code would look like:

```python
import torch
import torch.nn as nn
import torch.optim as optim

# Define a simple model (e.g., a small transformer - but we use a single linear layer for demonstration)
model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.1)
loss_fn = nn.MSELoss()

# Training loop for one step
input_tensor = torch.tensor([1.0])  # dummy input (in reality, it would be the tokenized instruction)
target_tensor = torch.tensor([1.0]) # target (probability of correct next word)

# Forward pass
prediction = model(input_tensor)
loss = loss_fn(prediction, target_tensor)

# Backward pass and update
optimizer.zero_grad()
loss.backward()
optimizer.step()
```

This does the same as our from-scratch code: it computes a prediction, compares to the target, computes the loss, and updates the model's weights using the gradient. `loss.backward()` automatically computes derivatives for all weights.


## What the code does

We start with a single weight and a training example. The `prediction` is the model's current output, and `target` is the desired output (1.0 for the correct word). We compute the loss as the squared difference. Then we compute a simplified derivative and update the weight: `weight = weight - learning_rate * derivative`. This is the core of gradient descent. In the PyTorch version, `loss.backward()` automatically computes the gradients for all trainable parameters, and `optimizer.step()` updates them. The loop would run over many examples (the whole dataset of instruction-response pairs) many times (epochs), gradually reducing the loss and making the model better at following instructions.


## Common misconceptions

**Instruction tuning teaches the model new facts** — Instruction tuning is like teaching the model new information, so it can answer questions about things it didn't know before.

Instruction tuning mainly changes behaviour — how the model responds to instructions — not the knowledge stored in the model. The model's knowledge comes from pretraining on a huge corpus of text. Instruction tuning doesn't add new facts; it teaches the model to present the facts it already knows in the form of an answer. If you want to add new information, you use retrieval (like RAG) or continued pretraining. In fact: Instruction tuning shapes how the model behaves (following instructions, answering questions) but does not significantly expand its knowledge. Knowledge is in the pretrained weights; instruction tuning refines the response style.

**More data is always better** — If I have a million instruction-response pairs, my instruction tuning will be much better than with 10,000 pairs.

Modern research shows that the *diversity* of instructions matters far more than the raw count. A small dataset with a wide variety of tasks and phrasings generalises better than a huge dataset with only a few types of questions. The model can overfit to the specific patterns in the data if they are repetitive. In fact: The quality and diversity of instruction-response pairs are more important than sheer quantity. A well-curated set of 50,000 varied instructions can outperform a million repetitive ones.

**I can skip instruction tuning and just use the base model** — The raw pretrained model already knows how to answer questions because it has read so much text.

The base model is a text-completion engine, not an assistant. If you ask it a question, it will continue the text in a way that might not be an answer at all. It has no incentive to follow instructions unless it has been trained on instruction-response pairs. In fact: Instruction tuning is essential to convert a base model into an assistant. Without it, the model doesn't understand the format of a question or command.

**Instruction tuning is only for chatbots** — My task is classification or text generation, so I don't need instruction tuning.

Instruction tuning is useful for any task where the model needs to follow a specific format or instruction. For classification, you can frame it as an instruction: 'Classify the sentiment of this review as positive, negative, or neutral.' The model learns to output the label. This is often more flexible than a fixed classification head. In fact: Instruction tuning is applicable to a wide range of tasks beyond chatbots, including classification, summarisation, translation, and code generation, by framing them as instruction-following problems.

**The model will never forget what it learned in pretraining** — Instruction tuning is safe because the model has learned so much already, it can't lose its general abilities.

This is exactly the danger of *catastrophic forgetting*. If the tuning dataset is very narrow or repetitive, the model can overwrite the general knowledge from pretraining, making it bad at anything outside the tuning data. In fact: Instruction tuning can cause catastrophic forgetting if not done carefully. Mixing in general data during tuning is a standard mitigation.

**The chat template doesn't matter** — As long as the instruction and response are in the data, the model will learn to follow instructions, regardless of the exact formatting.

The chat format template (system, user, assistant roles) is the standard representation that modern conversational models are trained on. Using the wrong template during inference can degrade performance drastically, even if the model was tuned properly. Consistency between training and inference is essential. In fact: The chat format template is critical. The model expects a specific structure, and using the wrong one can make the model produce poor responses.


## In short

Instruction tuning is the process of training a language model on instruction-response pairs to turn it from a text-completion engine into a helpful assistant. The key points are: (1) It changes behaviour, not knowledge; (2) diversity of instructions matters more than dataset size; (3) the chat format template must be consistent between training and inference; (4) the mathematics is just gradient descent — computing a loss and updating weights; (5) it is applied everywhere, from large models to small on-device models, RAG systems, and agentic tool-calling; and (6) it risks catastrophic forgetting if done poorly. Remember that a raw model just continues text — instruction tuning teaches it to respond to directions.


## Where this is used

Instruction tuning is the single most important step in making a language model usable as an assistant, and it appears in almost every modern AI system. Here's how it shows up across different applications:

**Large Language Models (LLMs)**: Every commercial model like ChatGPT, Claude, or Gemini goes through instruction tuning after pretraining. The pretrained model can generate text, but it doesn't know how to answer a question. Instruction tuning with thousands of hand-written or machine-generated instruction-response pairs teaches it to respond appropriately. For example, a user might ask 'Write a poem about a cat', and the instruction-tuned model knows to output a poem, not to continue with 'and a dog'. This is what makes the model 'conversational'. Without instruction tuning, these models would be far less useful — they would just complete the prompt.

**Small models running on devices (TinyML, on-device AI)**: Even small models like those in smart speakers or mobile keyboards use instruction tuning. A small language model on your phone might be trained to follow commands like 'Set a timer for 10 minutes' or 'What is the weather today?'. The device doesn't have the computing power for a huge model, so the model is compact, but it still needs instruction tuning to understand the format of a command and produce the right response. The tuning process is the same, just with a smaller dataset and a smaller model.

**Retrieval-Augmented Generation (RAG)**: In RAG systems, the model is given a user query plus a set of retrieved documents (e.g., from a database). The model must produce an answer based on those documents. Instruction tuning is crucial here because the model needs to learn to follow the pattern: 'Using the context provided, answer the question.' The instruction-tuned model knows to read the context and generate an answer grounded in it, rather than just continuing the text. For instance, if the retrieved context says 'The capital of France is Paris', and the user asks 'What is the capital of France?', the model should answer 'Paris'. Instruction tuning ensures the model understands this instruction-response-document relationship.

**Agentic systems (planning, tool calling, actions)**: Agents are AI systems that can take actions, like booking a flight or sending an email. They work by having the model generate a sequence of steps (a plan) and tool calls. Instruction tuning is what teaches the model to output structured actions like 'Call the weather API' or 'Send an email to [person] with subject [subject]'. The training data consists of instruction-response pairs where the response is a series of tool calls. Without instruction tuning, the model would just generate prose about booking a flight, not the actual API call. This is a fast-growing area, and instruction tuning on tool-use datasets is a common practice.

**Everyday applied machine learning**: Instruction tuning is used in many business applications: customer support chatbots, email auto-completion, code generation assistants, and more. A company might take a base model and instruction-tune it on their own support tickets to answer customer questions correctly. The developer would use a library like `transformers` from Hugging Face, load a pretrained model, and fine-tune it on their dataset of instruction-response pairs. The key is that the data must be formatted consistently (the chat format template) and the instructions must be diverse to generalise well.

**What breaks when done badly**: If instruction tuning is done with a small or non-diverse dataset, the model may overfit and only respond well to the exact instructions it saw, not to variations. If the template is inconsistent between training and inference, the model will be confused and give poor responses. If the tuning data contains mistakes or conflicting answers, the model will learn those errors. And if the learning rate is too high or too low, the model may forget what it learned during pretraining (catastrophic forgetting) or train very slowly. This is why the choice of data, format, and hyperparameters are so important.


---

Read this on the web: https://basavapurushottam.com/learn-ai/instruction-tuning
