Track 03 · Intermediate · ~35 min
The fastest compute is work you don’t repeat.
Nearly every serving optimization trades between compute and memory: stop wasting KV memory, reuse prefixes you’ve already computed, keep long prompts from stalling everyone else, and spend idle compute to emit more tokens per step.
PagedAttention: KV cache as pages.
Older servers reserved one contiguous KV region per request, sized for the maximum length. vLLM stores KV in fixed 16-token blocks allocated on demand, like OS virtual memory. Add requests and generate tokens on both, and compare.
Reserve-for-max vs allocate-on-demand
What
KV lives in fixed-size blocks (16 tokens by default). A per-request block table maps logical blocks to physical ones, so a request’s KV can be scattered anywhere.
Why it matters
The vLLM paper measured 60–80% of KV memory wasted by reserve-for-max allocation. Paging cut waste to under 4%, so batches got bigger and throughput jumped.
What blocks unlock
- Sharing prefixes (copy-on-write)
- Prefix caching by block hash
- Offloading and P/D transfer, block by block
Staff answer
“PagedAttention stores KV in fixed-size blocks through a per-request block table, which removed most fragmentation. The block then became the unit for everything else: prefix hashing, sharing, CPU offload and P/D transfer. Its costs are an extra indirection in the attention kernel and a block-size trade-off: small blocks mean finer reuse but tiny transfers.”
Prefix caching and the hash chain.
Each full block’s hash covers its parent’s hash plus its own tokens, so one hash identifies the entire prefix up to that point. Send two requests that share a system prompt, then break the chain with one tiny change.
Change one early token and every later block misses.
#39b918#a8338a#242bc2#e04c74#39b918#a8338a#242bc2#f85b7cWhat kills hit rates
- Timestamps or user IDs at the top of the prompt
- Reordered tool definitions
- Truncating context (shifts every block boundary)
- Routing that scatters a session across replicas
Design detail
Only full blocks are cached. Freed blocks go to an LRU free queue and stay reusable until evicted. V1 made this constant-time, costing under 1% throughput even at a 0% hit rate.
Security
A per-tenant cache_salt is mixed into the first block’s hash, so tenants never share blocks. That blocks timing attacks that probe other tenants’ prompts.
Staff answer
“Prefix caching reuses KV for shared prompt prefixes: block hashes chain through their parents, so a request walks its hashes and reuses the longest cached run. The operational work is protecting hit rate: stable prompt templates with dynamic content at the end, cache-aware routing, and hit rate as a first-class SLO signal.”
The token budget and head-of-line blocking.
Every GPU step has a token budget: decodes take 1 token each, prefills take chunks of what’s left. A 16,000-token document arrives while 32 users are decoding and short follow-up turns keep coming. Tune the budget and the per-request cap.
--max-num-batched-tokens · --long-prefill-token-thresholdWhat goes into each GPU step?
What
vLLM’s scheduler hands out {request: tokens} each step. Prompt and output tokens are treated alike, so chunked prefill falls out naturally.
The dial
A bigger budget finishes prefills sooner (better TTFT and throughput). A smaller budget bounds step time (better ITL). Pick the largest budget that meets your P99 ITL.
Real result
On agentic traffic (DeepSeek V4 Pro, B300), a 512-token long-prefill cap raised tokens per GPU-second by up to 93% and P90 interactivity by about 2.3×.
Staff answer
“The token budget is the main latency–throughput dial. With FIFO chunked prefill, one long prompt can take the whole budget step after step and block short turns. Capping per-request prefill tokens per step bounds its share. Priority scheduling alone doesn’t fix that, because a running prefill still consumes budget every step.”
Split prefill and decode onto different GPUs.
Prefill is compute-heavy, decode is bandwidth-heavy, and mixing them makes long prompts stall everyone’s stream. Disaggregation runs them in separate pools and ships the KV cache between them over RDMA.
Disaggregated: prefill pool → KV transfer → decode pool
How many prefill vs decode instances?
decode: 256 seqs ÷ 40 ms TPOT = 6,400 tok/s ÷ 500 = 12.8 req/s
→ 1.28 prefill instances per decode instance
What it buys
Separate tuning of TTFT and ITL, and no prefill interruptions for decodes. The vLLM docs say plainly that it doesn’t raise throughput on its own; the gains come from specializing each pool.
What it costs
KV transfer: 10K tokens of Llama-70B ≈ 3.3 GB ≈ 65 ms at 50 GB/s, unless pipelined layer by layer. Two pools to operate, RDMA networking, and a P:D ratio to keep matched.
When it’s worth it
Long prompts at scale, and MoE deployments where one prefill stalls a whole lockstep EP group. Not for short prompts or small fleets.
Staff answer
“Disaggregation separates prefill and decode into independently scaled pools, so each can use its own parallelism and kernels, and prefills stop interrupting decodes. The KV transfer becomes a networking problem, so I’d rate-match the pools with saturation sweeps and autoscale each on its own signal: queue and TTFT for prefill, KV usage and ITL for decode.”
Guess ahead, verify in one pass.
Decode is memory-bound, so checking k+1 positions costs about the same as generating one. A cheap drafter guesses k tokens; the target verifies them all at once and keeps the correct prefix. Output is identical to the target alone.
More tokens per expensive forward pass.
The maths
With acceptance rate α and k drafts, expected tokens per step = (1 − αk+1) / (1 − α). At α = 0.8, k = 4: about 3.4 tokens per pass. Each extra draft adds less.
When it hurts
At high load, batches are already compute-bound, so verification isn’t free. Llama3-70B on 4×H100: a draft model was 1.5× faster at QPS 1 but 1.4× slower at high QPS.
Drafters
Draft model · n-gram (copies from the prompt, great for RAG and code edits) · EAGLE-3 head · MTP heads · 2026 parallel drafters (P-EAGLE, DFlash, DSpark).
Staff answer
“Speculative decoding spends idle decode compute to emit several tokens per weight read, with rejection sampling keeping the output distribution identical. I’d gate it by load and route: enable it on latency-sensitive routes and the decode pool, watch acceptance rate per position, and benchmark TPOT at real concurrency, never only at QPS 1.”
Fewer bits, same answers (hopefully).
Smaller numbers mean less memory and less bandwidth. The right scheme depends on your bottleneck at your batch size, and every scheme must pass an accuracy gate on your own tasks.
What fits, and which kernel wins?
BF16 baseline
Best accuracy, but 141 GB means at least two H100s before any KV cache. The reference every quantized model is compared against.
accuracy baselineW4A16 · weight-only
Moves the fewest bytes per step. Mixed-input kernels (Marlin, Machete) dequantize in-kernel. Machete served Llama-3.1-70B on one H100 at 5 req/s with TTFT < 250 ms.
wins at batch 1: bandwidth-boundFP8 W8A8 · weights + activations
Runs on FP8 tensor cores at 2× BF16 peak FLOPs with no dequantize step. Wins when batches are big enough to be compute-bound.
runner-up hereWeight-only (W4A16)
4-bit weights, 16-bit maths. Moves the fewest bytes, so it’s best for memory-bound, small batches. Needs mixed-input kernels (Marlin, Machete).
Weights + activations (W8A8)
FP8 or INT8 on low-precision tensor cores doubles peak FLOPs. Best for compute-bound prefill and large batches. FP8 needs Hopper/Ada; Blackwell adds FP4.
Validate
- Compare against BF16 on your task mix
- Break out by task and context length
- Check tool-call and JSON validity
- Test FP8 KV separately (long context)
Staff answer
“I match the scheme to the bottleneck at the operating point: weight-only 4-bit for latency-bound small batches, FP8 W8A8 for throughput. Nothing ships without two gates on our own workload: accuracy against the BF16 baseline by task and context length, and performance at real concurrency.”
Reload or recompute?
KV cache is now a tiered resource: GPU memory, then CPU DRAM, local NVMe and shared pools (LMCache, Mooncake). When a prefix isn’t on the GPU, is it faster to fetch it or recompute it? Pick a tier and a prompt size.
Bytes over a wire vs FLOPs on a GPU
↑ faster · smaller ↓ bigger · slower
recompute ≈ 2 × 8B params × 10,100 tokens ÷ 500 TFLOPS (realistic H100 BF16; ignores attention’s T² term)
Rule of thumb
Transfer grows linearly with tokens; prefill compute grows linearly plus a quadratic attention term. So the longer the prompt, the more reloading wins.
Why DMA
vLLM’s offloading connector copies with cudaMemcpyAsync on the GPU’s DMA engines rather than a copy kernel, so transfers don’t steal SMs from the model: 83.4 vs 68.5 GB/s bidirectional on H100.
Payoff
CPU offloading cut single-request TTFT 2–22× and raised throughput up to 9× with many concurrent prefills. It pays when there’s reuse or preemption to save.
Staff answer
“I compare transfer time with prefill time. For Llama-8B, 10K tokens of KV is about 1.3 GB, roughly 26 ms from CPU memory, versus about 0.3 s to recompute, so reload wins by about 10× and the gap widens with context. Below a few hundred tokens fixed overheads dominate and recompute is simpler.”
Check yourself.
Production-flavoured questions. Get them wrong here, not in the interview.
1. What was PagedAttention’s main win?
2. After a deploy, prefix-cache hit rate fell from 85% to 5%. Most likely culprit?
3. Long document prompts make ITL spike for chat users on the same replicas. Which fix addresses it directly?
4. Speculative decoding was 1.5× faster at QPS 1 but 1.4× slower at high QPS. Why?
5. Batch-1 latency on a memory-bound decode: which quantization usually wins?
6. A 10K-token prefix of Llama-3.1-8B is in CPU memory. Reload or recompute?
0 / 6 answered