Lesson 8.1 · 23 min

Fine-Tuning: Adapting a Pre-Trained Model to Your Task

A model that has read half the internet still does not know how your support team answers a refund question, so how do we teach it that without starting from scratch?

In short: Fine-tuning takes a model that is already trained (pretrained) and continues training it on a small, focused dataset so it behaves the way our task needs. We reuse everything the model already knows and only nudge its weights. We can nudge all weights (full fine-tuning) or train a tiny add-on such as LoRA, which is far cheaper.

What is fine-tuning?

A model is a big function with millions or billions of adjustable numbers called weights (or parameters). Training means adjusting those weights so the model’s outputs match examples. Pretraining is the first, huge training run: a large language model (LLM) reads trillions of tokens of general text and learns grammar, facts, and reasoning patterns. The result is a base model or pretrained model.

Fine-tuning is a second, much smaller training run that starts from the pretrained weights and continues training on a dataset for one job. Instead of learning language from zero, the model only has to learn the difference between what it already does and what we want.

Think of it like hiring an experienced chef We do not teach a trained chef how to hold a knife. We hand them our restaurant’s recipe book and let them cook our menu for a week. Their general skill stays; they pick up our specific dishes and house style. Pretraining is culinary school; fine-tuning is the first week at our restaurant.

Our running example in this lesson: a support chatbot for a bike-rental shop. A general LLM can chat, but it does not know our refund rules, our friendly-but-short tone, or that it must always end with a booking link. We will fine-tune it on a few thousand past support conversations written the way we like.

Why do we need fine-tuning?

Pretrained models are generalists. They are good at many things and perfect at none of ours. There are three cheaper tools we should try first, and fine-tuning is for what they cannot fix:

• Prompting (writing clear instructions and examples in the input) changes behaviour for one request but costs tokens every time and can be ignored by the model.
• RAG (retrieval-augmented generation: fetching documents and pasting them into the prompt) gives the model fresh facts, but does not change its style or skills.
• Fine-tuning changes the weights, so the new behaviour is built in: a consistent format, tone, domain vocabulary, or a narrow skill like classifying tickets.

Typical wins from fine-tuning: shorter prompts (the instructions are “baked in”), more reliable output format (for example always valid JSON), better accuracy on a narrow task, and the chance to replace a big expensive model with a smaller fine-tuned one that is just as good on our task.

Pause and think: Our shop changes its prices every month. Should we fine-tune the prices into the model?

No. Prices are fast-changing facts. Fine-tuned knowledge is frozen in the weights and goes stale; we would need to retrain monthly. Put prices in a database and fetch them with RAG or a tool call. Fine-tune for stable things like tone, format, and how to handle a refund conversation.

How fine-tuning works, step by step

Fine-tuning uses the same machinery as ordinary training: a loss function (a number that measures how wrong the model is), gradients (for each weight, the direction that reduces the loss), and an optimizer (the rule that moves the weights, usually Adam or AdamW). The differences are where we start and how gently we move.

One fine-tuning run

1. Pick a base model: Choose a pretrained model whose size and licence fit our budget. For chat tasks we usually start from an instruction-tuned variant, not the raw base model.
2. Build the dataset: Collect input → ideal-output pairs, e.g. a customer message → the reply our best agent would write. Format them with the model’s chat template. Split off a validation set the model never trains on.
3. Load the pretrained weights: Training starts from these weights, not from random numbers. This is the whole point: we inherit the model’s knowledge.
4. Forward pass and loss: Feed a batch of examples, let the model predict the next token of the ideal reply, and compute cross-entropy loss. Usually the loss is computed only on the reply tokens, not on the prompt.
5. Backward pass and small update: Compute gradients and update weights with a small learning rate (often 10–100× smaller than in pretraining) so we adjust rather than overwrite.
6. Evaluate and stop early: After each epoch (one pass over the data), check validation loss and some real conversations. Stop when validation stops improving; usually 1–3 epochs.

A simple worked example with numbers

Let us shrink the idea to one weight so we can follow the arithmetic. Suppose a pretrained model has a weight w = 2.0 and, for an input x = 1, predicts ŷ = w · x = 2.0. Our task wants the answer y = 3.0.

• Loss (squared error): L = (ŷ − y)² = (2.0 − 3.0)² = 1.0.
• Gradient: ∂L/∂w = 2 · (ŷ − y) · x = 2 · (−1.0) · 1 = −2.0. Negative means “increase w”.
• Update with learning rate η = 0.1: w ← 2.0 − 0.1 · (−2.0) = 2.2.
• New prediction 2.2, new loss (2.2 − 3.0)² = 0.64. Lower than 1.0, so we moved the right way.

A real LLM does exactly this for billions of weights at once. Below is a slightly bigger, runnable version: a tiny “pretrained” classifier (a 4×3 weight matrix) adapted to a new task with only 60 examples. Notice it already starts at decent accuracy, because the pretrained weights are close to what the task needs.

fine_tune_tiny.py

import numpy as np
rng = np.random.default_rng(0)
# "Pretrained" model: a linear layer that already knows a general task
W_pre = rng.normal(size=(4, 3))            # 4 input features -> 3 classes
# Small task-specific dataset: 60 labelled examples for OUR task
X = rng.normal(size=(60, 4))
W_task = W_pre + rng.normal(scale=0.8, size=(4, 3))   # the task differs a bit
y = (X @ W_task).argmax(axis=1)
def softmax(z):
z = z - z.max(axis=1, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=1, keepdims=True)
def loss_acc(W):
p = softmax(X @ W)
loss = -np.log(p[np.arange(len(y)), y]).mean()   # cross-entropy
return loss, (p.argmax(axis=1) == y).mean()
W = W_pre.copy()                          # start FROM the pretrained weights
lr = 0.5                                  # small steps, few epochs
for epoch in range(31):
p = softmax(X @ W)
p[np.arange(len(y)), y] -= 1           # dLoss/dlogits for cross-entropy
W -= lr * X.T @ p / len(y)             # gradient step on every weight
if epoch % 10 == 0:
l, a = loss_acc(W)
print(f"epoch {epoch:2d}  loss {l:.3f}  accuracy {a:.0%}")
print("weights moved by", round(float(np.abs(W - W_pre).mean()), 3), "on average")

Output:

epoch  0  loss 0.513  accuracy 85%
epoch 10  loss 0.370  accuracy 92%
epoch 20  loss 0.310  accuracy 95%
epoch 30  loss 0.274  accuracy 97%
weights moved by 0.459 on average

Full fine-tuning vs LoRA

Full fine-tuning updates every weight. That is powerful but expensive. With the Adam optimizer in mixed precision, a common rule of thumb is about 16 bytes of GPU memory per parameter (2 for the weight, 2 for its gradient, and about 12 for the optimizer’s fp32 copy and two running averages), before counting activations. For a 7-billion-parameter model that is roughly 7 × 16 ≈ 112 GB, more than a single 80 GB GPU.

Parameter-efficient fine-tuning (PEFT) freezes the original weights and trains only a small number of new ones. The most popular PEFT method is LoRA (Low-Rank Adaptation): next to a frozen weight matrix W, it adds two thin matrices B and A and learns only those. The layer computes W·x + B·A·x. For a 4096 × 4096 matrix (about 16.8 million weights), rank-8 LoRA trains 2 × 4096 × 8 = 65,536 numbers, about 0.4%. The next lesson covers LoRA in depth.

When to use fine-tuning (and when not to)

A simple decision order works well in practice: prompt first, then RAG, then fine-tune. Each step is more expensive and slower to change than the one before.

Real-world uses Common fine-tuning jobs: ticket classification and routing, extraction of fields into strict JSON, writing in a brand voice, code completion for an internal codebase’s conventions, medical or legal summarisation styles, and distilling a big model’s behaviour into a small one for cheap serving. Hosted APIs from several vendors and open-source tools (for example Hugging Face transformers with peft) offer fine-tuning; exact options vary by vendor.

Tips before we fine-tune

• Quality beats quantity. A few hundred to a few thousand clean, consistent examples often beat a large noisy set. Every bad example teaches a bad habit.
• Match the real inputs. Training data should look like what users will actually send, including typos and short questions.
• Use a validation set and a baseline. Measure the base model with a good prompt first, so we know whether fine-tuning really helped.
• Keep the learning rate small and epochs few. Too much training memorises the dataset (overfitting) and erodes general skills.
• Use the model’s chat template. Formatting mismatches between training and serving quietly hurt quality.
• Test general skills afterwards. Check that the model still handles off-topic but reasonable questions politely.

Common mistake: catastrophic forgetting Training too long or with a high learning rate can make the model great at our task and noticeably worse at everything else. This is called catastrophic forgetting. Remedies: fewer epochs, lower learning rate, LoRA instead of full updates, and mixing in some general data.

Pause and think: Training loss keeps falling, but validation loss started rising after epoch 2. What is happening and what should we do?

The model is overfitting: memorising the training examples instead of learning the general pattern. Stop at the epoch with the best validation loss (early stopping), add more varied data, or lower the learning rate.

Common mistakes and how to spot them

A fine-tuning run rarely fails with an error message. It fails quietly, and the loss curves and a few sample outputs are our only clues. Before we read the curves, it helps to know how long the run really is. Take our bike-rental bot with 1,000 training examples, a batch size of 8 and 3 epochs (illustrative numbers).

Sizing the run with small numbers

1. Steps per epoch: One epoch shows every example once. 1,000 examples ÷ 8 per batch = 125 update steps per epoch.
2. Total steps: 3 epochs × 125 = 375 updates. That is all the “learning” there is. If the learning rate is too small, 375 tiny nudges may change almost nothing.
3. Hold some data back: We keep 100 of the 1,000 examples as a validation set and never train on them. Now an epoch is 900 ÷ 8 ≈ 113 steps, and we have an honest number to watch.
4. Decide when to look: We measure validation loss every 50 steps. That gives about 7 checkpoints, enough to see a trend and to pick the best one.

With that picture in mind, each failure has a recognisable shape:

One habit catches most of these early: before the full run, train on just 10 examples for a few dozen steps. The loss should fall close to zero. If it does not, something in the data format or the training setup is broken, and no amount of extra data will fix it.

Practice: try it yourself

We will fine-tune the smallest possible “model”: two weights. It starts out perfect at a general skill, and we fine-tune it on 20 examples of a slightly different task. We watch two numbers at once: the loss on our task, and the loss on the old general skill.

practice_fine_tuning.py

import numpy as np
rng = np.random.default_rng(0)
# General skill: y = 2*x1 + 1*x2. The "pretrained" weights already solve it.
X_gen = rng.normal(size=(200, 2))
y_gen = X_gen @ np.array([2.0, 1.0])
w_pre = np.array([2.0, 1.0])
# Our small task: only 20 examples, and its rule is a bit different.
X_task = rng.normal(size=(20, 2))
y_task = X_task @ np.array([2.0, 3.0])
def mse(w, X, y):
return float(np.mean((X @ w - y) ** 2))
def fine_tune(lr, steps):
w = w_pre.copy()                      # start from the pretrained weights
for _ in range(steps):
grad = 2 * X_task.T @ (X_task @ w - y_task) / len(y_task)
w -= lr * grad                    # one small nudge toward the task
return w
print("setting              task loss  general loss")
print(f"no fine-tuning       {mse(w_pre, X_task, y_task):9.3g}  {mse(w_pre, X_gen, y_gen):12.3g}")
for lr, steps in [(0.01, 20), (0.1, 5), (0.1, 100), (1.2, 20)]:
w = fine_tune(lr, steps)
name = f"lr={lr} steps={steps}"
print(f"{name:<20} {mse(w, X_task, y_task):9.3g}  {mse(w, X_gen, y_gen):12.3g}")

Output:

setting              task loss  general loss
no fine-tuning            5.02             0
lr=0.01 steps=20          1.55         0.658
lr=0.1 steps=5           0.361          2.08
lr=0.1 steps=100      8.34e-12          4.06
lr=1.2 steps=20       2.84e+22      1.42e+22

Read the table top to bottom. The more we fit the task, the more the general loss grows. That is forgetting, in two weights.

Now change it:

• Change the task rule on line 11 from [2.0, 3.0] to [2.0, 1.2], much closer to the pretrained weights. Predict first: will the general loss after 100 steps be higher or lower than 4.06?
• Add the setting (0.5, 20) to the list on line 25. Predict: does it converge like lr=0.1, or blow up like lr=1.2?
• Freeze the first weight by adding grad[0] = 0 after line 19. Predict: can the task loss still reach zero, and why does this particular task allow it?

Pause and think: In the output, lr=0.1 steps=100 has the best task loss and the worst general loss among the runs that did not blow up. If our bot must also keep its general skills, which row would we ship, and what extra data would help us decide?

Probably lr=0.1 steps=5 or a point near it: it removes most of the task loss (5.02 → 0.361) while roughly halving the damage to the general skill (2.08 instead of 4.06). To decide properly we need a validation set for the task and a small general test set, and we pick the checkpoint with the best balance. Training loss alone would always tell us to train longer.

Pause and think: The lr=1.2 run ends with a huge loss on both tasks. A teammate suggests training for more steps to recover. Will that work?

No. Each step overshoots the minimum by more than the previous one, so the weights move further away every time. More steps make it worse. The fix is a smaller learning rate, restarted from the pretrained weights, because the current weights are already ruined.

Summary

Fine-tuning is continued training from pretrained weights on a small, task-specific dataset. It changes behaviour (format, tone, narrow skills) reliably, while RAG is the better tool for changing facts. Full fine-tuning updates everything and costs the most; LoRA and other PEFT methods freeze the base and train a tiny add-on, which is how most teams fine-tune LLMs today. The craft is mostly in the data: clean examples, a held-out validation set, a small learning rate, and a check that general skills survive.

Key takeaways

• Fine-tuning continues training from pretrained weights on a small task dataset.
• Use it to change behaviour (format, tone, narrow skills); use RAG for fresh facts.
• Small learning rate, few epochs and a validation set prevent overfitting and forgetting.
• Full fine-tuning updates everything and is costly; LoRA trains under ~1% of weights.
• Try prompting first, then RAG, then fine-tuning.

Key terms

• Pretrained model: A model already trained on a large general dataset, used as the starting point.
• Fine-tuning: Continued training of a pretrained model on a smaller, task-specific dataset.
• Epoch: One full pass over the training dataset.
• Learning rate: How big each weight update step is; kept small during fine-tuning.
• PEFT: Parameter-efficient fine-tuning: freezing the base model and training a small number of new weights.
• Catastrophic forgetting: Losing previously learned skills when training on new data.
• Overfitting: Memorising training examples so performance on new data gets worse.

← 7.5 Jev and System One: Fast vs Deliberate AI Thinking · 8.2 LoRA: Parameter-Efficient Fine-Tuning via Low-Rank Matrices →