# Knowledge Distillation

Training a small student model to reproduce a large teacher's outputs, transferring capability into a cheaper package.

**Rests on:** Gradient Descent, Pre-Trained Model Adaptation

**Leads to:** Model Extraction

*LLMs and Generative AI · Inference Optimization*


## Why learn this

Knowledge distillation is the standard method for turning a massive, expensive artificial intelligence model into a small, fast version that can run on your phone or laptop. Without this technique, many modern AI tools would be too heavy to use outside of large data centers. By understanding this, you will see how companies make their powerful 'teacher' models available as lightweight 'student' versions for everyday devices. It also explains why so many smaller, openly available models exist: they are often distilled from larger giants to run locally on personal computers.


## The intuition

Imagine you are a new chef trying to learn how to make a perfect biryani. You have never cooked it before. You could read a recipe — that is like a dataset with hard labels: 'add 2 cups of rice', 'cook for 30 minutes'. But a recipe only tells you the steps, not the feel — the aroma, the color, the exact moment the rice is done.

Now imagine instead you stand next to a master chef who shows you their finished dish and tells you, 'This is what perfection looks like — aim for this.' You taste it, you see it, you smell it. You might learn faster and better than reading a recipe because you are not just copying steps, you are matching a richer target.

In artificial intelligence, a 'teacher' model is the master chef — big, slow, but incredibly skilled. It has seen millions of examples and learned complex patterns. When you train a model normally, you show it an image (say, a cat) and you tell it, 'The answer is cat'. That is a hard label — like a recipe. But the teacher model, when shown the same image, does not just say 'cat'. It also says, 'I am 98% sure it is a cat, 1.5% dog, 0.5% bird.' That soft distribution contains so much more information — it tells the student not just what the answer is, but what other answers were close. A cat picture that looks a bit like a dog teaches the student, 'These two are similar.' Hard labels never say that.

So knowledge distillation is simply: instead of teaching the student with the original data and hard labels, we teach it with the teacher's soft outputs. The student tries to mimic those outputs, and by doing so, it absorbs the teacher's wisdom — the relationships between classes, the subtle patterns — without needing to be as big or slow as the teacher.


## An analogy

The master craftsman and the apprentice. A master carpenter can build a beautiful chair, but she is slow and expensive, and she uses a huge workshop. The apprentice wants to learn to build the same chair, but quickly and with just a few hand tools.

If the apprentice just reads a list of measurements — 'the leg is 40 cm long, the seat is 50 cm wide' — she might make a chair that looks roughly right, but it will lack the subtle curves, the perfect balance, the feel of quality. The measurements are the hard labels: correct but limited.

So instead, the master sits the apprentice down and says, 'Watch me. I am going to build a chair, but I am also going to narrate my decisions: 'I am using this wood because it flexes slightly; I am rounding this edge because it feels better in the hand; I am using a dovetail joint here because it is stronger than a nail.' The apprentice does not just see what the master does — she hears the why behind the choices. This is the soft output: a rich description of the craft, not just the final product.

The apprentice then practices by trying to make chairs that would get the same praise from the master. Every time she makes a chair, she compares it to what the master would have made — not just the measurement, but the whole feel. Over time, the apprentice becomes nearly as good as the master, but she works with a much smaller set of tools and much less time.

Where the analogy breaks down: In our analogy, the apprentice learns why from the master's words. In real distillation, the student never hears any why. It only sees the teacher's output probabilities — soft numbers — and tries to get its own output numbers to match. It does not understand the reasoning; it just learns to produce the same result. Also, the apprentice gets better than the master over time (she might have her own style), but a distilled student is capped at the teacher's quality — it can only be as good as what it imitates.


## Definition

Knowledge distillation is a training method where a small model learns to copy the output probabilities of a larger model, transferring the larger model's learned knowledge into a more compact and efficient version.


## How it connects

This concept sits at the heart of inference optimization — the process of making models cheaper and faster to use. It works alongside quantization, which reduces the precision of numbers to save space, but distillation is different because it reduces the model's capacity by shrinking its size. A distilled model is often small enough that quantization alone is not enough; they are complementary steps. You will often see these used with local model deployment, which is the practice of running AI models on your own hardware rather than in the cloud.


## The mathematics

This concept rests on arithmetic — addition, multiplication, and comparisons — plus a dash of probability ideas (but only in words, not notation). It's not about calculus or algebra; it's about comparing two lists of numbers.

The core operation is the **weighted sum**. You have two lists of numbers of the same length. You multiply each pair of numbers together, add up all the products, and you get one number. That number tells you how similar the two lists are. If the lists are identical, the weighted sum will be its largest. If they are completely different, it'll be small. This is called a **dot product**, but you don't need that name yet.

Let's make it concrete. Suppose the teacher says 'cat: 0.98, dog: 0.01, bird: 0.01'. The student says 'cat: 0.90, dog: 0.05, bird: 0.05'. Are these similar? Let's compute the dot product:

- Multiply cat: 0.98 × 0.90 = 0.882
- Multiply dog: 0.01 × 0.05 = 0.0005
- Multiply bird: 0.01 × 0.05 = 0.0005
- Add: 0.882 + 0.0005 + 0.0005 = 0.883

Now imagine the student said 'cat: 0.50, dog: 0.25, bird: 0.25'. The dot product is 0.98×0.50 + 0.01×0.25 + 0.01×0.25 = 0.49 + 0.0025 + 0.0025 = 0.495 — much smaller. So the dot product gives us a number that says 'how much do these two lists agree?'.

So distillation is: we have the teacher's list (soft labels) and the student's list (its output). We compute the dot product (or a related measure) to see how close they are. Then we adjust the student's internal numbers (its weights) to make that dot product bigger — to make the student's list look more like the teacher's. That's the whole loop: compare, adjust, repeat.

There's also a special trick: we often divide the teacher's soft outputs by a 'temperature' before comparing, which makes them more spread out, revealing even finer similarities. That's just arithmetic: you divide each number by a constant. If the temperature is 1, you don't change anything. If it's 2, you divide each by 2 (so 0.98 becomes 0.49), and then you re-normalize so they add up to 1 again. The higher the temperature, the more the small probabilities (like 0.01) become relatively larger, teaching the student more about the 'almost' categories.

To measure similarity, we use the **dot product** of the two probability vectors:

similarity = Σᵢ teacherᵢ × studentᵢ

where teacherᵢ and studentᵢ are the probabilities for class i (e.g., cat, dog, bird).

In distillation, we also apply a temperature T to soften the distributions before comparison:

soft_teacherᵢ = exp(logitᵢ / T) / Σⱼ exp(logitⱼ / T)

soft_studentᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)

Then we train the student to maximize the dot product between these softened distributions (or equivalently minimize a distance between them).


## A worked example

```
The teacher model was trained on 1000 images of cats and dogs. For a particular image (which is actually a cat), the teacher outputs these probabilities: cat 0.90, dog 0.09, bird 0.01. The student model, untrained, sees the same image and outputs: cat 0.60, dog 0.30, bird 0.10. We use a temperature of 2 to soften the teacher's probabilities and then compare.

Step 1: First, soften the teacher's outputs by dividing by temperature T=2: cat 0.90/2 = 0.45, dog 0.09/2 = 0.045, bird 0.01/2 = 0.005.   # We divide each probability by the temperature to make the distribution 'softer' — less peaked, so smaller probabilities get relatively more weight.
Step 2: Now we re-normalize so they sum to 1: 0.45 + 0.045 + 0.005 = 0.5, so we divide each by 0.5: cat 0.90, dog 0.09, bird 0.01.   # After dividing, the numbers no longer sum to 1, so we scale them back so they form a valid probability distribution.
Step 3: Similarly, soften the student's outputs (temperature 2): cat 0.60/2 = 0.30, dog 0.30/2 = 0.15, bird 0.10/2 = 0.05. Sum = 0.5, so normalized: cat 0.60, dog 0.30, bird 0.10.   # The student's distribution is also softened; note that in this toy case, normalizing gives back the same numbers because the original distribution was already such that after halving, the sum was 0.5.
Step 4: Compute the dot product between the teacher's softened outputs (cat 0.90, dog 0.09, bird 0.01) and the student's softened outputs (cat 0.60, dog 0.30, bird 0.10): (0.90×0.60) + (0.09×0.30) + (0.01×0.10) = 0.54 + 0.027 + 0.001 = 0.568.   # We multiply each corresponding pair and add them up. This gives a single number indicating similarity.
Step 5: Now let's see what happens if the student were perfect. The teacher's distribution is (0.90, 0.09, 0.01). The dot product of the teacher with itself is (0.90×0.90)+(0.09×0.09)+(0.01×0.01) = 0.81 + 0.0081 + 0.0001 = 0.8182.   # This is the maximum possible similarity (when the student exactly matches the teacher). Our student's 0.568 is far from that, so we know it needs to adjust its parameters.

Result: The student's output is only 0.568 similar to the teacher's, while a perfect match would be 0.8182. The student will now adjust its internal numbers (weights) to increase this similarity, moving towards the teacher's distribution.
```


## From scratch, in Python

```python
# We'll implement a simplified distillation loop.
# This is not full training, just the core idea.

# A 'list' in Python is an ordered collection of items, like a shopping list.
# Here, we store the probabilities as lists of numbers.
teacher_probs = [0.90, 0.09, 0.01]   # teacher's output for cat, dog, bird
student_probs = [0.60, 0.30, 0.10]   # student's current output

temperature = 2.0

# Step 1: Soften the distributions using the temperature.
# We divide each number by T and then renormalize so they sum to 1.
def soften(probs, T):
    # First, divide each by T.
    softened = [p / T for p in probs]
    # Then compute the sum of the softened values.
    total = sum(softened)
    # Divide each by the total to make them sum to 1.
    normalized = [p / total for p in softened]
    return normalized

soft_teacher = soften(teacher_probs, temperature)
soft_student = soften(student_probs, temperature)

print("Soft teacher:", soft_teacher)
print("Soft student:", soft_student)

# Step 2: Compute the dot product to measure similarity.
def dot_product(list1, list2):
    # We start with a running total of 0.
    total = 0
    # Zip pairs up the items from both lists, e.g., zip([1,2],[3,4]) gives (1,3) then (2,4).
    for a, b in zip(list1, list2):
        total += a * b   # multiply the pair and add to the total
    return total

similarity = dot_product(soft_teacher, soft_student)
print("Similarity:", similarity)

# Step 3: In actual training, we would now adjust the student's weights to increase this similarity.
# But to illustrate, let's pretend we made the student a bit closer to the teacher:
student_probs_2 = [0.80, 0.15, 0.05]
soft_student_2 = soften(student_probs_2, temperature)
similarity_2 = dot_product(soft_teacher, soft_student_2)
print("Similarity after adjusting:", similarity_2)

# The loop would repeat: adjust, compute similarity, adjust again.
```


## The same thing, with a framework

```python
In practice, you would use a library like PyTorch. The dot product we computed by hand is just `torch.dot(soft_teacher, soft_student)`. The temperature scaling is done with a simple division and a softmax function. But understanding the plain Python loop is what it does underneath.
```


## What the code does

The code has four parts. First, we define the teacher's and student's probability lists. A list is a container that holds items in order — here, three numbers. Next, the `soften` function takes a list and a temperature, divides each number by T, then normalizes. Normalizing means dividing each by their sum so they add up to 1; this is what makes them 'probability-like'. We use a `for` loop to go through each element, and a `zip` loop in the dot product function to pair up corresponding elements from two lists. The `dot_product` function starts with a total of 0, then multiplies each pair and adds to total. The final part shows that by changing the student's probabilities (simulating weight adjustments), the similarity increases. In real training, the adjustment is driven by gradient descent, but we're just illustrating the comparison step.


## Common misconceptions

**Distillation is just training on the teacher's final answers** — Distillation just means I take the teacher's correct answers and train the student on those. It's the same as the original training.

The student would only learn the hard labels, missing the richer information in the probability distribution. The whole point is using soft targets — the probabilities — which encode similarity between classes (e.g., 'cat' is closer to 'dog' than to 'bird'). In fact: Distillation uses the teacher's probability outputs, not just the argmax. The soft distribution teaches the student 'how much' the teacher leans towards each class, which is far more informative than a single label.

**The student can surpass the teacher** — The student is learning from the best, so with enough training it could get even better than the teacher.

The student is trained to imitate the teacher's outputs. If the teacher makes a mistake, the student learns the mistake. The student's ceiling is the teacher's accuracy — it can't learn knowledge the teacher doesn't have. In fact: The student's performance is capped by the teacher's. That's why you usually distill from the largest, most accurate model you can find, even if it's too big to deploy.

**Temperature is a tuning knob for training speed** — Temperature just controls how fast the model learns, like a learning rate.

Temperature changes the shape of the probability distribution, which determines what information the student extracts from the teacher. Higher temperatures soften the distribution, making the student focus on the relative likelihood of all classes rather than just the top one. In fact: Temperature controls the 'softness' of the soft targets. A lower temperature (e.g., 1-3) keeps the distribution peaky; a higher temperature (e.g., 4-10) makes it flatter, exposing more of the teacher's 'almost' thinking. It's not a training speed control, but a balance between hard labels and full distribution.

**Distillation is only useful for classification** — Distillation works for image classification (cat/dog) but not for language models or other types of models.

Distillation works for any model that outputs a probability distribution over a set of possibilities — that includes language models, which predict the next token's probability distribution. You can also distill embeddings or intermediate features, not just final outputs. In fact: Distillation is general: it applies to classification, language modeling, sequence-to-sequence, and even to models that produce continuous outputs (like embeddings). The principle is the same — train the student to match the teacher's rich output distribution.

**The student doesn't need the original training data** — I have the teacher, so I can just generate outputs on any random noise and train the student. I don't need the original dataset.

The student needs to see representative examples to learn the task well. If you use random or unrepresentative data, the teacher's outputs are meaningless (it sees gibberish) and the student learns nothing useful. You need a dataset that covers the input distribution you care about. In fact: You need a large, diverse dataset — either the original training data or a similarly representative dataset. The teacher's soft labels on that data are what the student learns from. Using random noise is a waste of time.

**Distillation always preserves teacher quality perfectly** — Once I distill, the small model is just as good as the big one. The whole process is magic.

Distillation is a lossy compression. The student has fewer parameters, so there's a limit to how much it can capture. Some information is lost, and the student's quality is usually slightly below the teacher's. In fact: Distillation typically achieves 80-95% of the teacher's quality, not 100%. The goal is to get a small model that's 'good enough' for deployment, not perfect. You trade a small drop in accuracy for a huge drop in cost.


## In short

Knowledge distillation is a method to transfer the knowledge from a large, accurate teacher model into a small, fast student model. The student is trained to imitate the teacher's probability outputs (soft targets), not just the hard label. The key idea is that soft targets carry richer information — they reveal which classes are similar. The mathematics is just dot products and temperature scaling. This is how modern AI deploys models on phones and small servers: distill, then quantise, then deploy. Remember: the student can never exceed the teacher, temperature controls the softness of the targets, and distillation requires representative data.


## Where this is used

**Large Language Models (LLMs):** The most famous real-world example is distilling a massive model like GPT-4 (which has trillions of parameters) into a smaller model like GPT-4o-mini or other small models. Developers take the big model, run it on a huge dataset of prompts, and collect its outputs — not just the final answer, but the full probability distribution over the next token (the soft targets). Then they train the small model to predict exactly that distribution. This transfers the big model's ability to follow instructions, reason, and even its style, without needing the big model's size. The small model can then run on your phone or a small server.

**Small Language Models on Devices:** Smartphones and IoT devices have limited memory and battery. A 7-billion-parameter model might take 30 GB of RAM — too much. Distillation can shrink a 70B model to 7B, which fits in 4-8 GB after quantisation. The mechanism: the large model generates hundreds of thousands of text completions; the small model is trained to mimic those completions token by token. This is the workhorse behind on-device assistants like the ones in some modern phones.

**Retrieval-Augmented Generation (RAG):** In RAG, you need two models: one to convert text into embeddings (vectors) and another to generate answers. The embedding model is often a large transformer. You can distill the embedding model into a smaller one, so the retrieval step (which must be run on every query) becomes faster. For example, you train a large sentence-encoder to produce embeddings, then train a small student to produce the same embeddings for the same texts. The student is then used in production to embed millions of documents without cutting retrieval quality much.

**Agentic Systems:** Agents that plan and call tools often run on tight latency budgets because they loop many times — each loop needs a model call. If you have a powerful teacher model that plans well, you can distill its planning skill into a smaller student. The teacher generates a large set of 'thinking traces' — chains of reasoning that led to tool calls. The student is trained to reproduce those traces. This way, the student can make tool-calling decisions faster and cheaper, making the whole agent more responsive.

**Everyday Applied ML (vision, forecasting, fraud detection):** Anywhere you have a model that needs to run in real-time (like a fraud detection system that must score a transaction in milliseconds) you can build a large accurate model offline, then distill it into a small model for live scoring. In vision, a large CNN teacher can be distilled into a tiny network for mobile apps — for example, a model that recognizes plant diseases from a photo. In forecasting, a large model with many features can be distilled to a smaller one that uses fewer inputs, making it easier to deploy on edge devices.

**What a developer actually writes:** In practice, developers use libraries like PyTorch. They load a pretrained teacher (say, a Llama model), run inference on a large dataset to get the soft labels, then train the student with a loss function that measures the difference between the student's probabilities and the teacher's soft labels. The code is a few hundred lines. The hard part is data generation (getting the teacher's outputs) and tuning the temperature. If the temperature is too high, the student learns too much noise; too low, it learns only the hard labels. If they do it badly, the student might just memorize the teacher's mistakes, or the process might be so slow that it's not worth it. The guiding rule: the soft targets must be diverse and representative.


---

Read this on the web: https://basavapurushottam.com/learn-ai/knowledge-distillation
