Lesson 13.7 · 26 min

Continuous Batching: Keeping GPUs Busy Between Requests

Why should a user asking “What is 2+2?” have to wait for someone else's 1,000-word essay to finish before getting a GPU slot?

In short: GPUs serve LLMs efficiently only when many requests share each forward pass, so servers batch requests. Static batching keeps a batch together until its longest request finishes, leaving slots idle and newcomers waiting. Continuous batching (iteration-level scheduling) makes batch decisions at every decode step: finished requests leave immediately and waiting ones join, keeping the GPU full and raising throughput several-fold on realistic traffic.

The big picture

An LLM server receives a stream of requests of wildly different sizes: a one-line answer, a long summary, a code file. The GPU is expensive, so we want it doing useful work every millisecond. Batching (processing several requests in the same forward pass) is the main way to get there. The question is how to group requests when they all finish at different times.

Our running example is a support chatbot whose replies range from 5 tokens (“Yes, it is included.”) to 60 tokens or more (a step-by-step fix). We will compare two schedulers on the same 16 requests.

Quick recap: how an LLM generates tokens

Each request goes through prefill, one parallel pass over the prompt that builds its KV cache (the stored keys and values of past tokens), and then decode, one forward pass per output token. A reply of 60 tokens needs about 60 decode steps. Nobody knows in advance how many steps a request will need: it stops when the model emits an end-of-sequence token or hits a length limit.

The key fact for this lesson Decode work happens in iterations (steps). In each iteration, every request in the batch gets exactly one new token. That step-by-step structure is what continuous batching exploits.

Why batching matters for LLMs

A decode step for a single request is memory-bound: the GPU must stream all the model's weights (about 16 GB for an 8B model in FP16) from memory just to produce one token. The math units are mostly idle. If 32 requests share that pass, the weights are read once and used 32 times. Step time grows only a little, but tokens per step grow 32×.

So we want big batches. But a batch is only as useful as the number of slots doing real work. That is where the scheduling method matters.

The old way: static batching, and its problem

Static batching (also called request-level batching) works like this: collect up to B requests, run prefill for all of them, then run decode steps for the whole group until every request in it is finished. Only then start the next batch.

The ride-share analogy Static batching is a shuttle bus that leaves with 4 passengers and does not pick anyone up until it has dropped off the last passenger at the farthest stop. Seats empty out along the way, and people at the curb wait. Continuous batching is a ride-share van that picks up a new passenger the moment a seat frees up.

• Idle slots: a request that finishes after 6 tokens sits in the batch doing nothing (padding) until the 57-token request ends.
• Head-of-line blocking: new requests wait for the entire current batch to finish, even if they are tiny. Their time to first token suffers.
• Wasted padding work or memory: depending on the implementation, finished slots still consume compute or reserved memory.

Pause and think: In a static batch of 4 with output lengths 31, 33, 46 and 57, how many decode steps does the batch take, and how many slot-steps are idle?

It takes 57 steps (the longest). Total slot-steps are 4 × 57 = 228; useful ones are 31 + 33 + 46 + 57 = 167; so 61 slot-steps (27%) are idle.

What continuous batching is

Continuous batching, also called iteration-level scheduling or in-flight batching, makes the scheduling decision at every decode step instead of once per batch. The idea was introduced by the Orca system (Yu et al., OSDI 2022) and is now standard in vLLM, TensorRT-LLM, SGLang and Hugging Face TGI.

How continuous batching works, step by step

1. Check the batch: Before each iteration, the scheduler looks at the running requests and the waiting queue.
2. Evict finished requests: Any request that emitted end-of-sequence or hit its length limit leaves now. Its slot and KV cache memory are freed.
3. Admit waiting requests: If there are free slots and enough KV memory, new requests join. Their prompts need prefill, which is run in this iteration or split into chunks alongside decode work.
4. Run one iteration: One forward pass produces one new token for every decoding request in the batch.
5. Stream and repeat: Send new tokens to their users and go back to step 1. The batch composition changes continuously.

A numeric example

Let us simulate 16 chatbot requests with output lengths between 5 and 59 tokens on a GPU with 4 batch slots, ignoring prefill time for clarity. We count decode iterations and slot utilisation: the share of slot-steps that produce a useful token.

batching_sim.py

import numpy as np
rng = np.random.default_rng(1)
SLOTS = 4                                       # requests the GPU runs at once
lengths = rng.integers(5, 60, size=16).tolist() # output tokens each request needs
print("output lengths:", lengths)
def static_batching(lengths):
steps = 0
for i in range(0, len(lengths), SLOTS):     # take 4, run until ALL finish
steps += max(lengths[i:i + SLOTS])
return steps
def continuous_batching(lengths):
queue, running, steps = list(lengths), [], 0
while queue or running:
while queue and len(running) < SLOTS:   # refill free slots every step
running.append(queue.pop(0))
running = [r - 1 for r in running]      # one decode step for everyone
running = [r for r in running if r > 0] # finished requests leave now
steps += 1
return steps
useful = sum(lengths)                           # tokens we actually need
for name, fn in [("static", static_batching), ("continuous", continuous_batching)]:
steps = fn(lengths)
util = useful / (steps * SLOTS)
print(f"{name:10s}: {steps:4d} decode steps, slot utilisation {util:5.1%}, "
f"{useful/steps:.2f} tokens/step")

Output:

output lengths: [31, 33, 46, 57, 6, 12, 50, 57, 18, 22, 52, 28, 20, 50, 19, 27]
static    :  216 decode steps, slot utilisation 61.1%, 2.44 tokens/step
continuous:  152 decode steps, slot utilisation 86.8%, 3.47 tokens/step

The first static batch (31, 33, 46, 57) takes 57 steps; the second (6, 12, 50, 57) also takes 57, even though two of its requests finish within 12 steps. Continuous batching fills those freed slots right away. Notice also the effect on latency: the 6-token request in static batching cannot even start until step 57, while with continuous batching it starts as soon as any slot frees up.

Real numbers and speedup

Our toy had only 4 slots and moderate length variation. The gain grows with more slots and more variation in output lengths, because static batching's waste is driven by the gap between the longest and the typical request. Real chat traffic has long-tailed lengths, so the gap is large.

Published results vary by setup. The Orca paper reported large throughput improvements over the then-standard FasterTransformer at the same latency. A widely cited 2023 Anyscale benchmark reported up to about 23× throughput over naive static batching when continuous batching was combined with vLLM's memory optimisations; much of the largest gains come from the combination, not from scheduling alone. Treat any single number as workload-specific.

Benefits and important notes

• Higher throughput: slots stay busy, so more tokens per GPU-second.
• Lower queueing delay: new requests enter at the next iteration rather than after a whole batch.
• Fairer latency: short replies finish quickly instead of being held hostage by long ones.
• Better cost per token: the same hardware serves more users.

Prefill needs special handling A newly admitted request needs a prefill pass, which is much heavier than a decode step. Running it in the same iteration can delay everyone's next token. Engines handle this with chunked prefill (splitting the prompt into pieces mixed into several iterations) or by separating prefill and decode onto different GPUs.

Memory is the real limit Admitting a request means reserving room for its growing KV cache. Without paged KV memory, fragmentation limits how many requests can join; this is why continuous batching and PagedAttention are usually deployed together. If memory runs out mid-generation, the scheduler must preempt a request (pause it and later recompute or swap its cache back).

Common mistakes Assuming bigger batches are always better: each step gets slower as the batch grows, so per-user token speed (TPOT) rises; set a maximum batch size to protect latency targets. Also, continuous batching does not make a single request faster when the server is idle; it improves throughput and queueing under load.

Pause and think: Our server is nearly idle at night, with one request at a time. Will switching from static to continuous batching speed up those requests?

Not noticeably. With one request there is nothing to refill or interleave, so both schedulers behave the same. The gains appear under concurrent load with varied output lengths.

Worked example, step by step

The simulation gave us totals. Let us now trace a batch small enough to follow by hand. The GPU has 2 slots. Four requests are waiting at the start: A needs 6 tokens, B needs 2, C needs 1 and D needs 3. That is 12 useful tokens in total.

Reading the table

1. Static, first batch: A and B start together. B ends after step 2, but its slot stays idle for steps 3 to 6 because the batch is held until A ends.
2. Static, second batch: C and D can only start at step 7. C needs one token and then its slot idles for two more steps. Everything ends at step 9.
3. Continuous: When B leaves after step 2, C takes its slot at step 3. C leaves after one step and D takes the slot at step 4. Everything ends at step 6.
4. Count the slot-steps: Static used 9 steps × 2 slots = 18 slot-steps for 12 tokens: 67% utilisation. Continuous used 6 × 2 = 12 slot-steps for 12 tokens: 100%.
5. Look at each user: C finishes at step 7 under static batching and at step 3 under continuous batching. D finishes at step 9 instead of step 6. A and B see no change at all.

The last step shows who benefits: the requests that were waiting. A request that is already running gains nothing. This gives us a way to spot a scheduler that is not really continuous: if the queue is not empty and yet some slots are idle, requests are being held at a batch boundary.

Practice: try it yourself

The earlier simulation counted total decode steps. Here we look at the same idea from the user's side. Requests now arrive at different times, and we measure how long each one waits before it gets a slot. The request list is made up and tiny, so we can check every number by hand.

practice_batch_waits.py

# Per-request waiting time under static vs continuous batching.
# Each request is (arrival step, output tokens). The numbers are made up.
SLOTS = 2
requests = [(0, 6), (0, 2), (1, 1), (2, 3), (3, 1), (4, 4)]
def static(reqs):
t, start, finish = 0, {}, {}
for i in range(0, len(reqs), SLOTS):
group = range(i, min(i + SLOTS, len(reqs)))
t = max(t, max(reqs[j][0] for j in group))    # wait for the group to arrive
for j in group:
start[j], finish[j] = t, t + reqs[j][1]
t += max(reqs[j][1] for j in group)           # hold slots until the longest ends
return start, finish
def continuous(reqs):
t, nxt, running, start, finish = 0, 0, {}, {}, {}
while len(finish) < len(reqs):
while nxt < len(reqs) and len(running) < SLOTS and reqs[nxt][0] <= t:
running[nxt], start[nxt] = reqs[nxt][1], t   # admit into a free slot
nxt += 1
t += 1                                        # one decode step for everyone
for j in list(running):
running[j] -= 1
if running[j] == 0:                       # finished: leave right away
del running[j]
finish[j] = t
return start, finish
for name, fn in [("static", static), ("continuous", continuous)]:
start, finish = fn(requests)
waits = [start[j] - requests[j][0] for j in range(len(requests))]
print(f"{name:10s} wait before starting: {waits}  worst={max(waits)}  "
f"all done at step {max(finish.values())}")

Output:

static     wait before starting: [0, 0, 5, 4, 6, 5]  worst=6  all done at step 13
continuous wait before starting: [0, 0, 1, 1, 3, 2]  worst=3  all done at step 10

The third request needs a single token. Under static batching it waits 5 steps for a slot; under continuous batching it waits 1. Now change it:

• Set SLOTS = 3. Predict the worst wait for each scheduler before you run it. Which one gains more from the extra slot?
• Make every request the same length, for example 3 tokens, and keep the arrival times. Predict whether continuous batching still finishes earlier, and why.
• Change the first request to (0, 20). Predict what happens to the waits of the third and fourth requests under static batching, and whether continuous batching is affected in the same way.

Pause and think: In the practice output, continuous batching still makes one request wait 3 steps. The scheduler is working correctly, so what is the wait telling us?

Both slots were busy with real work when that request arrived, so the wait comes from too little capacity, not from idle slots. Continuous batching removes waiting caused by held slots. It cannot remove waiting caused by having more work than slots.

Pause and think: Suppose all requests arrive at step 0 and every one needs exactly 4 tokens. How do the two schedulers compare?

They behave the same. Every request in a batch ends on the same step, so no slot ever sits idle waiting for a longer neighbour, and both schedulers start the next group at the same time. The gain from continuous batching comes from differences in length and arrival time.

Quick summary

Batching amortises the cost of reading the model's weights across many users, which is what makes LLM decode affordable. Static batching wastes that opportunity by holding slots until the longest request ends. Continuous batching reschedules at every iteration, so finished requests leave and new ones join immediately. Combined with paged KV memory and chunked prefill, it is the backbone of modern serving engines.

Key takeaways

• Decode is memory-bound, so batching many requests into each forward pass multiplies throughput.
• Static batching holds a batch until its longest request ends, wasting slots and blocking newcomers.
• Continuous batching reschedules every iteration: finished requests leave, waiting ones join.
• Gains grow with batch size and output-length variance; in our toy, 216 → 152 steps for the same work.
• It relies on good KV memory management (paging) and careful prefill handling (chunking).

Key terms

• Batching: Processing several requests in the same forward pass to share the cost of reading weights.
• Static batching: Forming a batch once and running it until every request in it finishes.
• Continuous batching: Updating batch membership at every decode iteration; also called iteration-level or in-flight batching.
• Iteration: One forward pass that produces one new token for every decoding request in the batch.
• Slot utilisation: The share of batch slot-steps that produce useful tokens.
• Head-of-line blocking: New work waiting behind long-running work that it does not depend on.
• Preemption: Pausing a running request, freeing its memory, and resuming it later.

← 13.6 Paged Attention: OS-Inspired Memory Management for KV Caches · 13.8 Speculative Decoding: Draft Fast, Verify in Parallel →