Chapter 03

KV cache

Keep k and v. Drop q.

tokensThe␣capital␣of␣France␣isKV cache · one layer, one headkv

Same sequence, same layer. This time we keep what we compute.

← Attention, per decode step
01 / 10
Cache size, once

Per token, per sequence, the cache holds one kk and one vv for every head in every layer:

bytes per token=2⋅L⋅Hkv⋅dhead⋅bytes per value\text{bytes per token} = 2 \cdot L \cdot H_{kv} \cdot d_{head} \cdot \text{bytes per value}

Llama-3-8B: 2⋅32⋅8⋅128⋅2=131,0722 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072 bytes, so 128 KB per token, or about 1 GB for an 8k-token context. Grouped-query attention is why Hkv=8H_{kv} = 8 rather than the 32 query heads.