Lesson 8.4 · 24 min

Knowledge Distillation: Compressing Large Models into Small Ones

How can a small model learn more from a big model’s “wrong” answers than from the right answer alone?

In short: Knowledge distillation trains a small student model to imitate a large teacher model. Instead of learning only from hard labels (the one correct class), the student also matches the teacher’s full probability distribution, softened with a temperature, which carries “dark knowledge” about how classes relate. The result is a smaller, faster model that keeps much of the teacher’s quality.

What is knowledge distillation?

Knowledge distillation (KD) is a training method where a large, accurate model called the teacher passes what it knows to a smaller model called the student. The student is trained to reproduce the teacher’s outputs, not only the dataset’s labels. The idea was popularised by Hinton, Vinyals and Dean in 2015, building on earlier “model compression” work by Buciluǎ, Caruana and Niculescu-Mizil (2006).

Think of it like an apprentice An apprentice baker can learn from a recipe card that only says “correct: sourdough”. Or they can stand next to the master, who says “this dough is mostly sourdough, a bit like ciabatta, and nothing like a croissant”. The second kind of feedback teaches how things relate, and the apprentice learns faster from fewer loaves.

Running example: our bike-rental shop uses a large model to classify incoming support messages into categories like refund, damage, booking and spam. It is accurate but slow and costly. We want a small model that runs cheaply on every message, so we distil the big one into it.

Why we need knowledge distillation

Big models are usually more accurate, but they are expensive to serve: more memory, more latency, more energy, and sometimes too large for a phone or edge device. Training the small model directly on the labels often gives noticeably worse results than distilling it from a strong teacher. Distillation lets us get closer to big-model quality at small-model cost.

• Cheaper serving: fewer parameters means less GPU memory and lower cost per request.
• Lower latency: smaller models respond faster, which matters for real-time apps.
• On-device use: phones, browsers and embedded devices have tight memory budgets.
• Using unlabelled data: the teacher can label large amounts of raw data for the student.

Hard labels vs soft labels, and dark knowledge

A hard label is a one-hot answer: the correct class gets 1, all others 0. For an image of a cat, the hard label over [cat, dog, fox, car] is [1, 0, 0, 0]. It says nothing about which wrong answers are almost right.

A soft label is the teacher’s full probability distribution, for example [0.48, 0.26, 0.20, 0.07]. It still says “cat” most strongly, but it also says “this looks a bit like a dog and a fox, and hardly like a car”. Hinton and colleagues called this extra information dark knowledge: the relative probabilities of the wrong classes encode how the teacher sees similarity between classes.

Why does this help? Each soft label carries much more information than a hard label. The student learns that cats and dogs share features while cars do not, which helps it generalise from fewer examples and makes its mistakes more sensible.

Pause and think: For our support classifier, the teacher outputs refund 0.70, damage 0.25, booking 0.04, spam 0.01 for “the bike broke and I want my money back”. What dark knowledge does this carry?

That the message is mainly a refund request but strongly related to damage, and almost certainly not spam. A hard label (“refund”) would hide that damage is a close second, which is useful for the student to learn.

Temperature in the softmax

A model outputs raw scores called logits. Softmax turns logits into probabilities. A well-trained teacher is often very confident, so the wrong classes get tiny probabilities like 0.0003 that barely affect any loss. To reveal the dark knowledge, we divide the logits by a temperature T before softmax:

The student uses the same temperature when computing its own soft probabilities during distillation. After training, the student is used at T = 1 like any normal model. Typical temperatures in practice range from about 2 to 10; the best value is found by experiment.

The distillation loss

The classic distillation loss mixes two parts:

The soft loss makes the student match the teacher’s whole distribution. The hard loss keeps it anchored to the true label, which helps if the teacher is sometimes wrong. Many people write the soft term with cross-entropy instead of KL; the two differ only by a constant (the teacher’s entropy) that does not affect the student’s gradients.

distillation_loss.py

import numpy as np
classes = ["cat", "dog", "fox", "car"]
teacher_logits = np.array([6.0, 3.5, 2.5, -2.0])   # teacher's raw scores for one image
student_logits = np.array([3.0, 0.5, 0.0, 0.5])    # untrained-ish student
true_label = 0                                     # the hard label: "cat"
def softmax(z, T=1.0):
z = z / T
e = np.exp(z - z.max())
return e / e.sum()
for T in (1, 4):
p = softmax(teacher_logits, T)
print(f"teacher T={T}:", " ".join(f"{c}={v:.3f}" for c, v in zip(classes, p)))
T, alpha = 4.0, 0.7
p_t = softmax(teacher_logits, T)                   # soft targets
p_s = softmax(student_logits, T)                   # student at the same T
kl = np.sum(p_t * np.log(p_t / p_s))               # KL(teacher || student)
soft_loss = (T ** 2) * kl                          # T^2 keeps gradients comparable
hard_loss = -np.log(softmax(student_logits)[true_label])   # normal CE at T=1
total = alpha * soft_loss + (1 - alpha) * hard_loss
print(f"soft loss (T^2 * KL) = {soft_loss:.3f}")
print(f"hard loss (CE)       = {hard_loss:.3f}")
print(f"total = {alpha}*soft + {1 - alpha:.1f}*hard = {total:.3f}")

Output:

teacher T=1: cat=0.899 dog=0.074 fox=0.027 car=0.000
teacher T=4: cat=0.479 dog=0.256 fox=0.200 car=0.065
soft loss (T^2 * KL) = 1.332
hard loss (CE)       = 0.194
total = 0.7*soft + 0.3*hard = 0.991

Pause and think: In the output, the hard loss is small (0.194) but the soft loss is large (1.332). What does that tell us?

The student already gets the top answer right, so the hard label has little left to teach. The soft loss still sees a big gap: the student gives car too much and dog too little probability. Distillation keeps teaching where hard labels have gone quiet.

A step-by-step training walkthrough

Distilling our support classifier

1. Train or pick the teacher: Use the large, accurate model. It is frozen during distillation.
2. Choose a student: A smaller architecture, often the same family with fewer layers or a narrower width.
3. Run the teacher on the data: For each training message, get the teacher’s logits. This can be done once and cached (offline distillation).
4. Soften both outputs: Apply softmax at temperature T to the teacher’s and the student’s logits.
5. Compute the combined loss: α · T² · KL(teacher ‖ student) plus (1 − α) · cross-entropy with the true label.
6. Update the student only: Backpropagate and update the student’s weights. Repeat over many batches, then evaluate the student at T = 1.

Types of knowledge distillation

Distillation is a family of methods. They differ in what the student copies and when the teacher is trained.

• Offline: the teacher is pre-trained and frozen; the student learns afterwards. The most common setup.
• Online: teacher and student (or several peers) train at the same time and learn from each other.
• Self-distillation: a model teaches a new copy of itself, or deeper layers teach shallower ones.
• Sequence-level distillation for LLMs: the teacher generates whole answers, and the student is fine-tuned on them as ordinary text. This needs only the teacher’s text, not its logits, which is why it works even through an API.

Worked example, step by step

The combined loss can feel abstract, so let us compute the soft part by hand for the smallest possible case: two classes, refund and damage. The numbers are illustrative.

The soft loss for one message, by hand

1. Teacher at T = 1: Teacher logits are [2, 0]. Softmax gives e² / (e² + 1) ≈ 0.88 and 0.12. Quite confident.
2. Soften the teacher with T = 2: Divide the logits by 2: [1, 0]. Softmax gives ≈ 0.73 and 0.27. The second class is now much more visible.
3. Soften the student the same way: Student logits are [1, 0]. Divided by 2: [0.5, 0]. Softmax gives ≈ 0.62 and 0.38.
4. Compare the two distributions: KL = 0.73 · ln(0.73 / 0.62) + 0.27 · ln(0.27 / 0.38) ≈ 0.118 − 0.092 ≈ 0.027. Small, because the student already leans the right way.
5. Rescale by T²: T² · KL = 4 · 0.027 ≈ 0.11. This is the soft loss that enters the total.
6. Which way does the student move?: The teacher gives refund 0.73, the student only 0.62. The update raises the student’s refund logit and lowers damage, until the softened outputs match.

Now three failure cases, and what each looks like in practice:

A quick health check: compare student accuracy against the true labels with student–teacher agreement. High agreement with low accuracy means the teacher is the limit. Low agreement means the student or the training setup is.

Practice: try it yourself

We will train two tiny students on a single support message. One sees only the hard label “refund”. The other sees the teacher’s softened probabilities. Each student is just four logits, so we can watch exactly what it ends up believing about the wrong classes.

practice_distillation.py

import numpy as np
def softmax(z, T=1.0):
e = np.exp((z - z.max()) / T)
return e / e.sum()
classes = ["refund", "damage", "booking", "spam"]
teacher_logits = np.array([5.0, 3.0, 1.0, -2.0])   # teacher scores, one message
hard_label = np.array([1.0, 0.0, 0.0, 0.0])        # the true class: refund
def train_student(use_teacher, T=3.0, steps=300, lr=0.5):
z = np.zeros(4)                                # student logits start flat
for _ in range(steps):
if use_teacher:   # gradient of T^2 * KL(teacher(T) || student(T))
grad = T * (softmax(z, T) - softmax(teacher_logits, T))
else:             # gradient of cross-entropy with the hard label
grad = softmax(z) - hard_label
z -= lr * grad
return softmax(z)                              # the student is used at T = 1
t, h, s = softmax(teacher_logits), train_student(False), train_student(True)
print("class      teacher  hard-only  distilled")
for i, c in enumerate(classes):
print(f"{c:<10} {t[i]:7.3f}  {h[i]:9.3f}  {s[i]:9.3f}")
# How does each student order the three wrong classes?
gap = lambda p: p[1] / p[3]                        # damage vs spam
print(f"damage is {gap(t):.0f}x more likely than spam for the teacher")
print(f"damage is {gap(h):.0f}x more likely than spam for the hard-only student")
print(f"damage is {gap(s):.0f}x more likely than spam for the distilled student")

Output:

class      teacher  hard-only  distilled
refund       0.866      0.995      0.866
damage       0.117      0.002      0.117
booking      0.016      0.002      0.016
spam         0.001      0.002      0.001
damage is 148x more likely than spam for the teacher
damage is 1x more likely than spam for the hard-only student
damage is 148x more likely than spam for the distilled student

Both students pick “refund”. Only the distilled one learned that a broken-bike message is far closer to “damage” than to “spam”. The hard-only student treats all wrong classes the same.

Now change it:

• On line 11, change the default steps=300 to steps=5. Predict: after only 5 updates, does either student already rank “damage” well above “spam”? Which one, and why?
• Change T=3.0 on line 11 to T=1.0. Predict: will the distilled student still match the teacher in the end? What changes is the speed, not the target. Check with steps=20.
• Make the teacher wrong: set teacher_logits on line 8 to [3.0, 5.0, 1.0, -2.0]. Predict the distilled student’s top class. What does this say about trusting the teacher blindly?

Pause and think: The hard-only student ends at 0.995 for “refund”, higher than the teacher’s 0.866. Is that student “better” than the teacher?

No. It is only more confident, because a one-hot target keeps pushing the correct class toward 1.0 for as long as we train. It has thrown away everything about how the other classes relate. On messages that sit between two classes, that overconfidence makes it less useful, not more.

Pause and think: The student trained at T = 3 but its printed probabilities match the teacher’s T = 1 probabilities to three decimals. How can that be?

Matching the softened distributions at T = 3 means the student’s logits divided by 3 match the teacher’s logits divided by 3, up to a constant. So the logits themselves match up to a constant, and softmax ignores constants. If the student matches at one temperature and has enough capacity, it matches at every temperature, including T = 1.

Real examples, pitfalls and wrapping up

In LLM products Many vendors offer small, fast model tiers, and distillation from larger models is a common ingredient, though details are often not public. Teams also distil privately: use a large model to label or answer thousands of in-domain examples, then fine-tune a small open model on them. Always check the teacher’s licence and terms of use before training on its outputs.

Common mistakes Using T = 1 so the soft targets are almost one-hot and add little; forgetting the T² factor so the soft term barely influences training; choosing a student so small it cannot represent the task (the “capacity gap”); and assuming the student can beat the teacher’s mistakes. The student inherits the teacher’s biases and errors along with its knowledge.

When not to distil: if the small model trained directly already meets the quality bar, or if serving cost is not a constraint, distillation adds work for no gain. And if no good teacher exists for the task, there is nothing to distil.

Wrapping up: knowledge distillation compresses a big model into a small one by training the student on the teacher’s soft probabilities. Temperature exposes the dark knowledge in those probabilities, the T²-scaled KL term teaches it, and a hard-label term keeps the student honest. Variants copy features or relations, and for LLMs the simplest form is fine-tuning a small model on text generated by a large one.

Key takeaways

• Distillation trains a small student to imitate a large teacher’s outputs.
• Soft labels carry dark knowledge: how wrong classes relate to the right one.
• Temperature T > 1 flattens softmax so that knowledge becomes learnable; the student uses the same T.
• Loss = α·T²·KL(teacher‖student) + (1−α)·CE(label), with T² keeping gradients balanced.
• Variants copy outputs, features or relations; for LLMs, fine-tuning on teacher-generated text is common.

Key terms

• Teacher: The large, accurate model whose behaviour is being transferred.
• Student: The smaller model trained to imitate the teacher.
• Soft label: A full probability distribution over classes, as opposed to a single correct class.
• Dark knowledge: The similarity information hidden in a model’s probabilities for incorrect classes.
• Temperature: A divisor applied to logits before softmax; higher values flatten the distribution.
• KL divergence: A measure of how one probability distribution differs from another.

← 8.3 Prefix Tuning: Learnable Context Prepended to the Input · 8.5 Continual Learning: Training Without Forgetting the Past →