an illustrated guide · 21 of 29 chapters

Learning SGLang

Inference engines, from the problems up.

Written and drawn by Wilson Zheng.

Start at chapter 1 →Star on GitHub
The capital of France is▍the prompt, then one token per turn of the loopforwardpickappenduntil EOS
Every chapter is one picture like this, stepped with ← →, ending on the question the next chapter answers.

The loop

Why every piece of the engine exists.

  1. 01Bare-minimum inferenceOne function, called in a loop.
  2. 02Attention, per decode stepWhat a new token needs from the past.
  3. 03KV cacheKeep k and v. Drop q.
  4. 04Prefill vs decodeTwo very different workloads.
  5. 05Batching & continuous batchingSharing a GPU between users.
  6. 06The schedulerTwo lists, one GPU, one choice per step.
  7. 07KV memoryPages, and the memory cap.
  8. 08Prefix cachingSame beginning, one copy.
  9. 09One request, end to endFrom an HTTP POST to the GPU and back.
  10. 10Scheduler deep diveOne Req through one step.

Making one GPU fast

The loop is correct; now the step is slow for reasons that have nothing to do with the model.

  1. 11CUDA graphsLaunching a thousand kernels as one.
  2. 12Piecewise CUDA graphsWhen the graph has to break.
  3. 13torch.compileFusing the kernels you were about to launch.
  4. 14Attention backendsWhy there are five kernels for one equation.
  5. 15Speculative decodingGuess several tokens, verify in one step.

Beyond text · SGLang Omni

Speech and vision break the assumptions of Part I, one at a time.

  1. 16Omni modelsPixels and speech in, speech out.
  2. 17Stages and schedulersSeven stages, three kinds of scheduler.
  3. 18Between stagesEight edges, four ways to cross them.
  4. 19Sharing GPUsSeveral models, one card.
  5. 20Streaming audio decodeOutput with a real-time deadline.
  6. 21Full duplexListening while speaking.

More from one GPU

Every pass still reads all of the weights: make that read smaller, and keep the GPU busy while it waits.

  1. 22QuantizationShrinking the read every step pays for.soon
  2. 23Two-batch overlapHiding communication behind compute.soon

More than one GPU

The model, the cache, or the traffic no longer fits on one card.

  1. 24Tensor parallelismA model bigger than one GPU.soon
  2. 25Pipeline & expert parallelismSplitting by layer, and by expert.soon
  3. 26Data-parallel attentionReplicate the attention side, share the rest.soon
  4. 27Hierarchical KV cacheWhen 64 GB is not enough.soon
  5. 28Prefill-decode disaggregationTwo clocks, two machines.soon
  6. 29The routerMany engines, one front door.soon