Chapter 07

KV memory

Pages, and the memory cap.

one H100 · 80 GB of HBMweights 16 GB36k48k32k40kroom for KV cache · 64 GBin use · 31 GB, 8 requestsempty right now · 33 GBeach running request grows 128 KB per step; the empty part is what newcomers can be admitted intoSGLang keeps about 17% of the card for activations and CUDA graphs (--mem-fraction-static, 0.83 here), so its pool is nearer 55 GB

The scheduler checks for room for KV cache before admitting a new request. 80 GB on the card, 16 GB of weights, and everything else is room for KV cache. Running requests fill it a token per step.

← The scheduler
01 / 07
The memory arithmetic, once

Per token, the cache costs 2⋅L⋅Hkv⋅dhead⋅bytes2 \cdot L \cdot H_{kv} \cdot d_{head} \cdot \text{bytes}, 128 KB for Llama-3-8B in bf16. The batch cap is

Bmax⁡=⌊HBM−weightscontext×128 KB⌋B_{\max} = \left\lfloor \frac{\text{HBM} - \text{weights}}{\text{context} \times 128\,\text{KB}} \right\rfloor

SGLang does not hand the cache all of HBM−weights\text{HBM} - \text{weights}: it sizes the pool to f⋅HBM−weightsf \cdot \text{HBM} - \text{weights}, where ff is --mem-fraction-static (about 0.83 for this model on one H100), and keeps the rest for activations and CUDA graphs. That makes the pool about 55 GB here; the stage leaves ff out.

Reserving the maximum length per request wastes whatever is never written, typically well over half, and fragments as requests finish. Paging with page size pp wastes at most p−1p - 1 tokens per request and never fragments: a request’s cache is a list of page numbers, and the attention kernel gathers through that list. In SGLang, MHATokenToKVPool holds the K and V, TokenToKVPoolAllocator hands out free slots (page size 1 by default), and req_to_token holds the page tables.

Admission (PrefillAdder) counts more than the pages a step writes. It sets aside each running request’s expected remaining output, min⁡(max_new_tokens−written, 4096)\min(\text{max\_new\_tokens} - \text{written},\ 4096) times a ratio that starts at 0.7 and decays to about 0.1 as requests finish early, and it charges a newcomer its prompt plus min⁡(max_new_tokens, 4096)\min(\text{max\_new\_tokens},\ 4096). Retraction is a separate check: when the running batch’s own next step does not fit and nobody is finishing, retract_decode sends back the request with the fewest output tokens, to be prefilled again later. Sharing pages across requests is the next chapter.