Chapter 17

Stages and schedulers

Seven stages, three kinds of scheduler.

where do Ch. 16 Step 4's models run? · SGLang has one process with a model in itSGLang today · Ch. 9 Step 5server + tokenizerCPU · no modelscheduler + modelGPU · one loopdetokenizerCPU · no modelqueuequeueQwen3-Omni's six models · Ch. 16 Step 4vision encoderaudio encoderthe enginetalkercode predictorcode2wavrun inside the scheduler process:the encoders in its prefill (Ch. 16 Step 2)no process runs theseneeded: somewhere for every model to run, each in the right kind of loop, joined by queueswhite: one pass · blue: a loop (Ch. 16 Step 4) · a red dashed outline: no process runs it in SGLang

Today, SGLang runs one model in one process: the scheduler, which also runs the encoders inside its prefill. The talker, code predictor and code2wav have no process to run in.

← Omni models
01 / 12
Where this chapter’s claims come from

SGLang-Omni. The stages, their schedulers and their defaults are read from SGLang-Omni’s Qwen3-Omni pipeline and its design notes (docs/design/refactor_rfc.md).

The code predictor’s cache. The code predictor keeps a cache like any decoder, but a short one: the talker’s hidden state, code 0 and the codes it picks, 16 positions, and it starts over every frame. There is nothing in it for a stage’s scheduler to keep between steps, which is a second reason it lives inside the talker’s step.

The schedulers, by name. One pass is SimpleScheduler; it batches whatever is already waiting (up to 32 requests for each encoder) and by default does not wait for more. A loop is OmniScheduler, which calls SGLang’s own Scheduler methods on its own state and overrides only the ones that touch the inbox or the model runner; the talker’s QwenTalkerScheduler adds the ready check on top. Streaming is StreamingDetokenizeScheduler for decode and Code2WavScheduler for code2wav, each written for its stage.

What the talker switches off. Its configuration turns off overlap scheduling, the radix cache and chunked prefill. The reasons on the stage for the first two are this book’s reading of the code: each talker step’s input is the frame the step before wrote, and its prompt arrives as vectors rather than token ids. SGLang-Omni’s code does not say why chunked prefill is off, so the stage only notes that it is.

The detokenizer’s example. How Qwen3-Omni’s tokenizer splits a particular emoji into tokens is not shown here; the stage only uses one to show a character arriving in two pieces.