Lesson 4.11 · 23 min

Causal Masking: Preventing the Model from Seeing the Future

If a model is trained to predict the next word, but its attention can see the whole sentence, what stops it from simply peeking at the answer?

In short: A causal mask blocks every token from attending to tokens that come after it. We set the scores above the diagonal of the attention matrix to −∞ before softmax, so those weights become exactly 0. This lets a decoder-only LLM train on every position of a sentence in parallel without cheating, and makes training match how text is generated at inference time: one token at a time, left to right.

The problem: a next-word predictor that can see the future

A decoder-only LLM (the GPT, Llama, Claude family of models) is trained on one simple task: given the tokens so far, predict the next token. From the sentence "The cat sat down" we get several training examples at once: after "The" predict "cat", after "The cat" predict "sat", after "The cat sat" predict "down".

Transformers process all positions in parallel: the whole sentence goes through attention as one matrix. That is great for speed, but it creates a danger. In plain self-attention, every token can attend to every other token, including the ones to its right. When the model sits at position "cat" and tries to predict the next word, attention would let it look straight at "sat", which is the answer.

Think of it like an exam with the answer sheet face up A student taking a fill-in-the-blank test with the answer key visible will score perfectly while learning nothing. On the real day, with no answer key, they fail. A causal mask turns the answer sheet face down: each position may only read what came before it.

At generation time there is no future to look at: the model writes tokens one by one, and the next token does not exist yet. If training let the model rely on future tokens, it would learn a skill it can never use when it matters. The causal mask forces training to match inference. "Causal" here means "respecting the order of cause and effect in time": earlier tokens can influence later ones, never the other way around.

Without causal masking

Recall the attention recipe: scores = Q·Kᵀ / √dₖ, then softmax each row, then multiply by V. Let us take some illustrative scaled scores for "The cat sat down" and apply softmax without any mask. Each row shows how much one token attends to each token.

Look at the first row. The token "The" puts 63% of its attention on "down", the last word. If this representation is used to predict the word after "The", the model is reading ahead. During training, the loss would quickly drop to near zero by copying future tokens, and the model would learn nothing useful about language.

Pause and think: In the unmasked matrix, which row is already "legal" (it uses only past and present tokens)?

The last row, "down". Every token is at or before position 4, so nothing it attends to is in the future. Every other row leaks some weight to tokens on its right.

With causal masking

With a causal mask, row i may only use columns 1 to i. We take the same scores, replace every entry above the diagonal with −∞ (minus infinity), and run softmax again.

Three things to notice:

• The first token can only attend to itself, so its weight is 100%.
• The remaining weights are renormalized: the probability that used to go to the future is shared among the allowed tokens in proportion to their scores. Row "cat" was 17% / 47% and now becomes 27% / 73%.
• The last row is unchanged, because it had no future tokens to hide.

The causal mask matrix

The mask itself is just a fixed pattern that depends only on the sequence length T. There are two common ways to write it:

Why −∞ and not 0? Because softmax exponentiates: e^(−∞) = 0 exactly, so a masked position gets weight 0 no matter what its score was. Setting a score to 0 would not hide it: e⁰ = 1, which is a perfectly ordinary weight. The allowed region (j ≤ i) is a lower-triangular pattern including the diagonal, which is why you will also hear "triangular mask" or "look-ahead mask".

Implementation of causal masking

Causal attention, step by step

1. Compute scores: scores = Q·Kᵀ / √dₖ, a T × T matrix for T tokens.
2. Build the mask: Take a T × T matrix of ones and keep only the strict upper triangle (above the diagonal). In numpy: np.triu(ones, k=1).
3. Apply it: Wherever the mask is true, replace the score with −∞ (or add the additive mask).
4. Softmax each row: Masked entries become exactly 0; each row's remaining weights sum to 1.
5. Mix values: Multiply the weights by V as usual. Each output now depends only on its own token and earlier tokens.

causal_attention.py

import numpy as np
np.set_printoptions(precision=2, suppress=True)
rng = np.random.default_rng(42)
tokens = ["The", "cat", "sat", "down"]
T, d = 4, 8
Q, K, V = (rng.standard_normal((T, d)) for _ in range(3))
def softmax(z):
z = z - z.max(axis=-1, keepdims=True)
e = np.exp(z)                       # exp(-inf) = 0
return e / e.sum(axis=-1, keepdims=True)
def attend(Q, K, V, causal):
scores = Q @ K.T / np.sqrt(d)
if causal:
mask = np.triu(np.ones((T, T), dtype=bool), k=1)  # True above diagonal
scores = np.where(mask, -np.inf, scores)
w = softmax(scores)
return w, w @ V
mask = np.triu(np.ones((T, T), dtype=bool), k=1)
print("mask (1 = blocked):\n", mask.astype(int))
w, out = attend(Q, K, V, causal=True)
print("causal weights:\n", w)
# Leak test: change the LAST token's key and value, keep everything else
K2, V2 = K.copy(), V.copy()
K2[3] += 5.0
V2[3] += 5.0
for causal in (False, True):
_, a = attend(Q, K, V, causal)
_, b = attend(Q, K2, V2, causal)
changed = [tokens[i] for i in range(T) if not np.allclose(a[i], b[i])]
print(f"causal={causal}: outputs that changed -> {changed}")

Output:

mask (1 = blocked):
[[0 1 1 1]
[0 0 1 1]
[0 0 0 1]
[0 0 0 0]]
causal weights:
[[1.   0.   0.   0.  ]
[0.56 0.44 0.   0.  ]
[0.36 0.3  0.34 0.  ]
[0.19 0.44 0.18 0.19]]
causal=False: outputs that changed -> ['The', 'cat', 'sat', 'down']
causal=True: outputs that changed -> ['down']

The leak test is the most useful way to convince yourself (or a unit test) that masking works: perturb a future token and check that earlier outputs do not move.

Pause and think: What would go wrong if we wrote np.triu(..., k=0) instead of k=1?

k=0 includes the diagonal, so each token would be blocked from attending to itself. The first row would then have every entry at −∞, softmax would divide 0 by 0 and produce NaN. The diagonal must stay allowed.

Causal vs bidirectional attention

Real-world use, pitfalls and limits

Where causal masks show up Every decoder-only LLM applies a causal mask in every self-attention layer during training and during the prefill step at inference (when the whole prompt is processed at once). Attention libraries usually expose a flag such as is_causal=True so the kernel can skip the masked half of the work entirely instead of building a T × T matrix. The causal structure is also what makes the KV cache valid: since earlier tokens never depend on later ones, their keys and values never need recomputing when a new token arrives.

In real batches, the causal mask is often combined with a padding mask: shorter sequences are padded with filler tokens to a common length, and those filler positions must also be hidden. The two masks are simply combined so a position is blocked if either mask blocks it.

Common mistakes Masking after softmax instead of before (rows stop summing to 1 and the future leaks through the denominator). Using k=0 and masking the diagonal (NaN rows). Using a huge negative number like −1e9 in float16, which overflows; use −∞ or the dtype's minimum value. Forgetting the mask when writing custom attention, which shows up as a training loss that drops suspiciously fast and a model that generates nonsense.

When not to use it: if the task is understanding a complete input (classifying a support ticket, producing an embedding for search), a causal mask throws away useful right-hand context. Bidirectional encoders are usually a better fit there. And note what causal masking does not do: it does not tell the model where tokens are. Position information comes from separate mechanisms such as RoPE, covered in a later lesson.

Going one level deeper

We said the causal mask is often combined with a padding mask. Let us see exactly how, on one short sentence. “The cat sat” has 3 tokens, but its batch needs length 5, so two filler tokens are added on the right. Two rules now apply at once: a token may not look at the future, and nobody may look at filler.

Building the combined mask

1. Causal part: Block every cell above the diagonal. This depends only on the length, 5.
2. Padding part: Block every column that belongs to a filler token. This depends on the data: here columns 4 and 5.
3. Combine: A cell is blocked if either rule blocks it. In code this is a logical OR of two true/false tables.
4. Softmax: Blocked cells get −∞ and end up with weight 0. Each row shares its weight among what is left.
5. Mask the loss too: The filler positions still produce a prediction. We leave those out of the loss, so the model is never trained to predict filler.

Notice a subtle point. With filler on the right, the three real tokens already could not see it: it lies in their future, so the causal mask hides it. The padding mask only changes the filler rows. The part that truly protects training here is leaving filler out of the loss.

Where padding goes wrong Put the filler on the left and the picture changes. The first row is a filler token that may see only itself, and itself is blocked. The whole row is −∞, softmax divides 0 by 0, and the result is NaN. One NaN then spreads through every later layer. If a model suddenly outputs NaN only for batches with mixed lengths, a fully blocked row is the first thing to look for.

Practice: try it yourself

We will build the combined mask for a padded sentence, turn it into attention weights, and then list for every position what it can see, what it must predict and whether that prediction counts in the loss.

practice_causal_padding.py

import numpy as np
np.set_printoptions(precision=2, suppress=True)
# One short sentence, padded to length 5 so it fits in a batch
tokens = ["The", "cat", "sat", "<pad>", "<pad>"]
T = len(tokens)
is_pad = np.array([t == "<pad>" for t in tokens])
causal = np.triu(np.ones((T, T), dtype=bool), k=1)   # True = future token
padding = np.tile(is_pad, (T, 1))                    # True = filler column
blocked = causal | padding                           # blocked if either says so
print("blocked (1 = hidden):\n", blocked.astype(int))
# Equal scores everywhere, so the weights show only the effect of the masks
scores = np.where(blocked, -np.inf, np.zeros((T, T)))
e = np.exp(scores - scores.max(axis=1, keepdims=True))
weights = e / e.sum(axis=1, keepdims=True)
print("weights:\n", weights)
# Training targets: the input shifted left by one position
targets = tokens[1:3] + ["<end>", "<pad>", "<pad>"]
for i in range(T):
sees = [tokens[j] for j in range(T) if not blocked[i, j]]
in_loss = "yes" if targets[i] != "<pad>" else "no (ignored)"
print(f"{tokens[i]:5s} sees {str(sees):22s} predicts {targets[i]:5s} in loss: {in_loss}")

Output:

blocked (1 = hidden):
[[0 1 1 1 1]
[0 0 1 1 1]
[0 0 0 1 1]
[0 0 0 1 1]
[0 0 0 1 1]]
weights:
[[1.   0.   0.   0.   0.  ]
[0.5  0.5  0.   0.   0.  ]
[0.33 0.33 0.33 0.   0.  ]
[0.33 0.33 0.33 0.   0.  ]
[0.33 0.33 0.33 0.   0.  ]]
The   sees ['The']                predicts cat   in loss: yes
cat   sees ['The', 'cat']         predicts sat   in loss: yes
sat   sees ['The', 'cat', 'sat']  predicts <end> in loss: yes
<pad> sees ['The', 'cat', 'sat']  predicts <pad> in loss: no (ignored)
<pad> sees ['The', 'cat', 'sat']  predicts <pad> in loss: no (ignored)

Now change it:

• Change blocked = causal | padding to blocked = causal. Predict first: which rows of the weights change, and which stay exactly the same?
• Change k=1 to k=2 in np.triu. Predict what “The” can now see, and why that ruins training.
• Move the filler to the left: tokens = ["<pad>", "<pad>", "The", "cat", "sat"]. Predict the first row of the weights (ignore the target printout for this one).

Pause and think: The filler rows print weights of 0.33 over the three real tokens. So the model does real attention work for filler positions. Does this change anything for the real tokens, and is it free?

It changes nothing for the real tokens: no row may attend to a filler column, and the filler predictions are left out of the loss. But it is not free. Those rows still cost compute and memory in every layer. That is why training pipelines group sentences of similar length into the same batch, to keep filler small.

Pause and think: The position “sat” must predict <end>, and that prediction counts in the loss. What would the model fail to learn if we ignored it, the way we ignore filler?

It would never learn when to stop. The end marker is a real token with a real meaning: the text is complete. A model that was never trained to predict it would keep generating until it hits the length limit. Filler has no meaning and is skipped; the end marker must be learned.

Key takeaways

• A causal mask lets each token attend only to itself and earlier tokens.
• It is applied by adding −∞ to scores above the diagonal before softmax, giving those positions exactly zero weight.
• It lets one forward pass train next-token prediction on every position in parallel without cheating.
• Test it with a leak test: changing a future token must not change earlier outputs.
• Encoders like BERT skip it; decoder-only LLMs need it in every self-attention layer.

Key terms

• Causal mask: A triangular mask that blocks attention from any token to tokens after it.
• Decoder-only model: A Transformer that generates text left to right using causally masked self-attention.
• Bidirectional attention: Attention where every token can see tokens on both sides.
• Lower-triangular matrix: A matrix whose entries above the diagonal are zero (or, here, blocked).
• Padding mask: A mask that hides filler tokens added to make sequences in a batch the same length.
• Prefill: The inference step where the whole prompt is processed in parallel before generating new tokens.

← 4.10 Scaled Dot-Product Attention: Why We Divide by √dₖ · 4.12 Multi-Head Attention: Many Perspectives at Once →