an illustrated guide · 13 of 27 chapters

Learning SGLang

Inference engines, from the problems up.

Written and drawn by Wilson Zheng.

Start at chapter 1 →Star on GitHub
One picture per chapter. Each chapter is a single visualization you step through. The captions are the narration; there is almost no other text.
Every chapter ends on a question. The next chapter is the answer. By the end, an inference engine isn't a pile of features but the obvious response to a chain of "so what do we do about that?"
Keys. ← → to step, space to play. The last step of a chapter offers the next one.

The loop

Why every piece of the engine exists.

  1. 01Bare-minimum inferenceOne function, called in a loop.
  2. 02Attention, per decode stepWhat a new token needs from the past.
  3. 03KV cacheKeep k and v. Drop q.
  4. 04Prefill vs decodeTwo very different workloads.
  5. 05Batching & continuous batchingSharing a GPU between users.
  6. 06The schedulerTwo lists, one GPU, one choice per step.
  7. 07KV memoryPages, and the memory cap.
  8. 08Prefix cachingSame beginning, one copy.
  9. 09One request, end to endFrom an HTTP POST to the GPU and back.
  10. 10The scheduler's loopOne Req through one step.

Making one GPU fast

The loop is correct; now the step is slow for reasons that have nothing to do with the model.

  1. 11CUDA graphsLaunching a thousand kernels as one.
  2. 12Piecewise CUDA graphsWhen the graph has to break.
  3. 13torch.compileFusing the kernels you were about to launch.
  4. 14Attention backendsWhy there are five kernels for one equation.soon
  5. 15Speculative decodingGuess several tokens, verify in one step.soon
  6. 16QuantizationShrinking the read every step pays for.soon
  7. 17Two-batch overlapHiding communication behind compute.soon

More than one GPU

The model, the cache, or the traffic no longer fits on one card.

  1. 18Tensor parallelismA model bigger than one GPU.soon
  2. 19Pipeline & expert parallelismSplitting by layer, and by expert.soon
  3. 20Data-parallel attentionReplicate the attention side, share the rest.soon
  4. 21Hierarchical KV cacheWhen 64 GB is not enough.soon
  5. 22Prefill-decode disaggregationTwo clocks, two machines.soon
  6. 23The routerMany engines, one front door.soon

Beyond text · SGLang Omni

Speech and vision break the assumptions of Part I, one at a time.

  1. 24Multimodal inputsA stage in front of the loop.soon
  2. 25Multi-stage inferenceThinker, talker, and the pipe between them.soon
  3. 26Streaming audio decodeOutput with a real-time deadline.soon
  4. 27Full duplexListening while speaking.soon