Chapter 05

Batching & continuous batching

Sharing a GPU between users.

memoryweights16 GBread 16 GBonce per stepcompute1 × 16 GFLOPbusy 0.3% of the step1 user, one token eachtok1tokens per step4.8 msstep time0.3%GPU busy209tokens per second

One user. One token per step, 4.8 ms per step, and the GPU mostly waiting.

← Prefill vs decode
01 / 09
The batching arithmetic, once

With BB sequences decoding together, one step costs max⁡(tmem,  B⋅ttok)\max(t_{\text{mem}},\; B \cdot t_{\text{tok}}), so throughput is

Bmax⁡(tmem,  B ttok)  tokens/s\frac{B}{\max(t_{\text{mem}},\; B\,t_{\text{tok}})} \;\text{tokens/s}

linear in BB until B≈n∗≈300B \approx n^* \approx 300, then flat. Static batching pays for that with idle slots and queueing; continuous batching (Orca, 2022) re-forms the batch at every step, which is why every serving engine does it now.