Lesson 6.4 · 23 min

Sliding Window Attention: Taming Very Long Contexts

If each token may only look at its last few thousand neighbours, how can a model still connect ideas that are a hundred thousand tokens apart?

In short: Sliding window attention lets each token attend only to the previous W tokens instead of the whole sequence. That turns attention cost from quadratic to linear in sequence length and caps the KV cache at W entries per layer. Because layers are stacked, information can still hop W tokens per layer, so distant tokens are reachable indirectly, and many modern models mix sliding-window layers with a few full-attention layers.

What is attention?

Attention is how a Transformer lets each token gather information from other tokens. Every token produces a query (what it is looking for), a key (what it offers) and a value (what it hands over). The token compares its query with every allowed key, turns the scores into weights with softmax, and takes a weighted average of the values.

In a chat LLM, attention is causal: a token can look at itself and earlier tokens, never later ones. Which pairs are allowed is described by a mask, a grid where row i, column j says whether token i may look at token j.

The problem with normal attention

With full causal attention, token number n compares itself with all n earlier tokens. Over a sequence of length n that is about n²/2 scores per head per layer. Double the length and the work roughly quadruples.

Generation has a second cost. The model keeps the keys and values of every past token in the KV cache. With full attention that cache grows with every token, forever. For a 100,000-token document it can need many gigabytes per user, and every new token has to read all of it.

Think of it like reading a long novel Full attention is re-reading every previous page before writing each new word. Sliding window attention is keeping only the last few pages open on the desk. You still remember the plot from earlier chapters, because those chapters shaped the pages you are looking at now.

Running example: a customer-support log of 50,000 tokens. The current question is about a refund, and the customer's order number appeared 30,000 tokens ago. We will see how each kind of attention copes with that gap.

What is sliding window attention?

Sliding window attention (SWA) restricts each token to a fixed-size window of the most recent tokens. With window size W, token i may attend to tokens i − W + 1 through i. As i moves forward, the window slides with it.

Cost per token is now at most W scores, so total work grows linearly: about n × W instead of n²/2. The KV cache per layer never needs more than W entries; older entries can be overwritten in a rolling buffer, a fixed-size circular array where position i is stored at slot i mod W.

A simple step-by-step walkthrough

Take 8 tokens and a window of W = 3. Token 5 (counting from 0) may look at tokens 3, 4 and 5 only.

What happens for token t7

1. Build the query: t7 projects its vector into a query.
2. Find the window: With W = 3, only t5, t6 and t7 are allowed. t0–t4 are masked.
3. Score: Compute 3 dot products (query · key) instead of 8, and divide by √dₖ.
4. Softmax: Turn the 3 scores into weights that sum to 1. Masked tokens get weight 0.
5. Mix values: The output for t7 is the weighted average of the values of t5, t6, t7.
6. Evict: When t8 arrives, t5 leaves the window and its cache slot is reused for t8.

How information still travels far away

At layer 1, t7 only sees t5–t7. But t5's layer-1 output already mixed in t3 and t4. So at layer 2, when t7 reads t5, it indirectly receives information from t3. Each layer extends the reach by about W − 1 positions. After L layers, the receptive field (how far back information can come from) is roughly L × W tokens.

Mistral 7B (2023) used W = 4,096 with 32 layers, giving a theoretical reach of about 32 × 4,096 = 131,072 tokens even though each layer looks only 4,096 back.

sliding_window.py

import numpy as np
def sliding_mask(n, w):
# token i may attend to token j if j <= i (causal) and i - j < w (window)
i = np.arange(n)[:, None]
j = np.arange(n)[None, :]
return (j <= i) & (i - j < w)
n, w = 8, 3
m = sliding_mask(n, w)
for row in m.astype(int):
print(" ".join("x" if v else "." for v in row))
print("scores computed: full causal =", n * (n + 1) // 2, "| sliding =", int(m.sum()))
# How far back can information reach after stacking layers?
reach = np.eye(n, dtype=bool)                  # layer 0: each token knows itself
for layer in range(1, 4):
reach = (m.astype(int) @ reach.astype(int)) > 0   # one more hop through the window
print(f"after {layer} layer(s), last token sees tokens from position",
int(np.argmax(reach[-1])))
# Mistral-7B-style numbers: window 4096, 32 layers
W, L = 4096, 32
print("theoretical reach:", W * L, "tokens")
print("KV cache kept per layer: full grows with n, sliding stays at", W)

Output:

x . . . . . . .
x x . . . . . .
x x x . . . . .
. x x x . . . .
. . x x x . . .
. . . x x x . .
. . . . x x x .
. . . . . x x x
scores computed: full causal = 36 | sliding = 21
after 1 layer(s), last token sees tokens from position 5
after 2 layer(s), last token sees tokens from position 3
after 3 layer(s), last token sees tokens from position 1
theoretical reach: 131072 tokens
KV cache kept per layer: full grows with n, sliding stays at 4096

Theoretical reach is not real memory Information from far away must survive many hops, getting mixed and diluted at each layer. In practice a pure sliding-window model is much weaker at exact long-range recall (like quoting the order number from 30,000 tokens ago) than full attention. "Can reach" does not mean "reliably remembers".

Pause and think: A model uses W = 1,024 and 24 sliding-window layers. Roughly how far back can information travel in theory? Could it reliably copy an exact number that appeared 20,000 tokens earlier?

About 24 × 1,024 ≈ 24,600 tokens in theory, so 20,000 is just inside the reach. But reliable exact copying is unlikely: the number must be relayed through about 20 layers of mixing. That is why many models keep some full-attention layers.

Comparing normal attention and sliding window attention

Where sliding window attention is used

• Longformer (2020) combined a sliding window with a few global tokens that attend everywhere, for long-document understanding.
• Mistral 7B (2023) popularised SWA in a decoder LLM, with a 4,096-token window and a rolling-buffer KV cache.
• Gemma 2 and Gemma 3 alternate local sliding-window layers with global full-attention layers; Gemma 3 uses several local layers per global one to cut KV memory at long context.
• gpt-oss (2025) alternates dense attention layers with small banded (sliding) window layers.
• Streaming setups combine a sliding window with "attention sink" tokens, covered in the next lesson.

Back to the support log In a hybrid model, the many local layers handle grammar and nearby context in the 50,000-token log cheaply, while the few global layers can directly fetch the order number from 30,000 tokens earlier. A pure SWA model with a 4,096 window would have to relay it through many layers and may lose the exact digits.

Advantages and trade-offs

• Advantage: linear cost. Work and memory grow with n × W rather than n², making long inputs affordable.
• Advantage: bounded memory. The KV cache per layer is fixed, so memory use is predictable for any conversation length.
• Advantage: locality matches language. Most dependencies in text are local, so little is lost on many tasks.
• Trade-off: weaker far recall. Exact retrieval of a distant fact is harder, which hurts "needle in a haystack" style tasks.
• Trade-off: window choice matters. Too small hurts quality; too large removes the savings.
• Trade-off: eviction effects. Simply dropping the oldest tokens during streaming can destabilise the model because of attention sinks; the next lesson explains why and how to fix it.

Pause and think: We must build a contract-review tool that quotes exact clause numbers from anywhere in 200-page contracts. Should we use a pure sliding-window model?

Probably not. Exact long-range recall is the weak spot of pure SWA. Prefer a model with full-attention layers (or a hybrid with global layers), possibly with retrieval to fetch relevant clauses.

Worked example, step by step

Two questions come up whenever we design with sliding windows: how many layers does a fact need to travel a given distance, and how much cache does a mix of local and global layers really save? Let us work both out with small numbers.

Relaying a fact 9 tokens back with W = 4

1. Layer 1: The fact sits at token 0. Token 9 sees only tokens 6 to 9, so it sees nothing of it. But token 3 sees tokens 0 to 3 and mixes the fact into its own output.
2. Layer 2: Token 6 sees tokens 3 to 6. It reads token 3's layer-1 output, which now carries a blend that includes the fact.
3. Layer 3: Token 9 sees tokens 6 to 9. It reads token 6's layer-2 output. The fact has arrived, after 3 hops and 3 rounds of mixing.
4. The rule: Layers needed = ⌈distance / (W − 1)⌉. Here ⌈9 / 3⌉ = 3.
5. Real scale: In our support log the order number is 30,000 tokens back. With W = 4,096 that is ⌈30,000 / 4,095⌉ = 8 layers of relaying. A single full-attention layer does it in 1 hop.

The hybrid keeps about 23% of the full cache. Notice where that cache goes: the 2 global layers hold 100,000 of the 140,960 entries. So in a hybrid model, the number of global layers decides the memory bill, and the window size matters much less.

Practice: try it yourself

We will build the rolling buffer cache from this lesson in plain Python. Each new token overwrites the slot i mod W. We print which token gets evicted and which positions the new token can still see.

practice_rolling_buffer.py

# A rolling-buffer KV cache: the entry for position i lives in slot i mod W
W = 4
buffer = [None] * W
tokens = "order 7419 arrived late so please refund it".split()
for i, tok in enumerate(tokens):
slot = i % W
evicted = buffer[slot][1] if buffer[slot] else "-"
buffer[slot] = (i, tok)                      # overwrite the oldest entry
# the token at position i can attend to whatever is in the buffer now
visible = sorted(pos for pos, _ in (b for b in buffer if b))
print(f"i={i} {tok:<8} slot={slot} evicts={evicted:<8} sees positions {visible}")
# Work and memory for the whole sequence
n = len(tokens)
full = n * (n + 1) // 2
slide = sum(min(i + 1, W) for i in range(n))
print(f"n={n}: full causal scores={full}, sliding scores={slide}")
print(f"cache entries per layer: full={n}, sliding={W}")
in_cache = [tok for _, tok in buffer]
print("still cached:", in_cache, "| is '7419' cached?", "7419" in in_cache)

Output:

i=0 order    slot=0 evicts=-        sees positions [0]
i=1 7419     slot=1 evicts=-        sees positions [0, 1]
i=2 arrived  slot=2 evicts=-        sees positions [0, 1, 2]
i=3 late     slot=3 evicts=-        sees positions [0, 1, 2, 3]
i=4 so       slot=0 evicts=order    sees positions [1, 2, 3, 4]
i=5 please   slot=1 evicts=7419     sees positions [2, 3, 4, 5]
i=6 refund   slot=2 evicts=arrived  sees positions [3, 4, 5, 6]
i=7 it       slot=3 evicts=late     sees positions [4, 5, 6, 7]
n=8: full causal scores=36, sliding scores=26
cache entries per layer: full=8, sliding=4
still cached: ['so', 'please', 'refund', 'it'] | is '7419' cached? False

Now change it:

• Set W = 8. Predict the sliding score count and whether 7419 is still cached at the end.
• Set W = 2. Predict which token evicts 7419, and the new sliding score count, before running.
• Add four more words to the sentence. Predict the slot of the last token and the list of cached tokens at the end.

Pause and think: In the output, please evicts 7419 at i = 5. Can a later token in this layer still be influenced by the order number?

Not directly: its key and value are gone from this layer's cache. Indirectly, yes. Tokens 1 to 4 attended to it while it was cached, so their outputs carry a blend of it, and later tokens can read those outputs in the next layer. That is the relay effect, and it is why far-back details arrive blurred.

Pause and think: Why store position i in slot i mod W instead of shifting every entry one place to the left when a new token arrives?

Shifting copies W entries for every new token; the modulo trick overwrites one entry in place. The order of entries inside the buffer does not matter, because attention treats them as a set and position information is carried separately. Same visible tokens, far less memory traffic.

Key takeaways

• SWA limits each token to the last W tokens, turning n² attention cost into about n × W.
• The KV cache per layer is capped at W entries and can live in a rolling buffer.
• Stacking L layers lets information travel about L × W tokens, but only indirectly.
• Exact long-range recall is SWA's weak spot; hybrids add full-attention layers to compensate.
• Used in Longformer, Mistral 7B, Gemma 2/3 and gpt-oss, often mixed with global layers.

Key terms

• Sliding window attention: Attention where each token sees only the most recent W tokens.
• Window size (W): How many recent tokens, including itself, a token may attend to.
• Attention mask: A grid marking which query-key pairs are allowed; blocked pairs get weight 0.
• Receptive field: How far back information can reach a token after stacking layers.
• Rolling buffer cache: A fixed-size KV cache where new entries overwrite the oldest, using slot i mod W.
• Global attention layer: A full-attention layer mixed into a mostly local model to restore long-range access.

← 6.3 Grouped Query Attention: Fewer KV Heads, Same Quality · 6.5 Attention Sinks: The Hidden Cost of Extended Context →