Chapter 07

KV memory

Pages, and the memory cap.

one H100 · 80 GB of HBMweights 16 GB36k48k32k40kroom for KV cache · 64 GBin use · 30 GB, 8 requestsempty right now · 34 GBeach running request grows 128 KB per step; the empty part is what newcomers can be admitted into

The scheduler checks for room for KV cache before admitting a new request. 80 GB on the card, 16 GB of weights, and everything else is room for KV cache. Running requests fill it a token per step.

← The scheduler
01 / 07
The memory arithmetic, once

Per token, the cache costs 2⋅L⋅Hkv⋅dhead⋅bytes2 \cdot L \cdot H_{kv} \cdot d_{head} \cdot \text{bytes}, 128 KB for Llama-3-8B in bf16. The batch cap is

Bmax⁡=⌊HBM−weightscontext×128 KB⌋B_{\max} = \left\lfloor \frac{\text{HBM} - \text{weights}}{\text{context} \times 128\,\text{KB}} \right\rfloor

Reserving the maximum length per request wastes whatever is never written, typically well over half, and fragments as requests finish. Paging with page size pp wastes at most p−1p - 1 tokens per request and never fragments: a request’s cache is a list of page numbers, and the attention kernel gathers through that list. In SGLang, TokenToKVPool is the page pool (page size 1 by default), req_to_token holds the page tables, and retract_decode is the eviction when nothing frees up. Sharing pages across requests is the next chapter.