Omni models
Pixels and speech in, speech out.
Text gets a tokenizer in front of the engine and a detokenizer after it (Ch. 1 Steps 2 and 11). An omni request needs a converter on each side too: a photo and a spoken question in, a spoken answer out.
The running example, and where its numbers come from
Qwen3-Omni. Chapters 16 to 21 follow one model, Qwen3-Omni, as SGLang-Omni serves it. It reads text, images, audio and video, and answers in text or in speech.
A sample. Sound is stored as the height of the wave, read at a fixed rate. Qwen3-Omni reads speech at 16,000 samples a second and writes it at 24,000, the rates SGLang-Omni’s server takes and returns.
What the stage simplifies, and where each claim comes from
The codec. Qwen3-Omni describes each 80 ms frame of sound with 16 codes of 2,048 values each, and its code-to-sound model writes 1,920 samples per frame (24,000 a second ÷ 12.5 frames). The talker picks code 0; a small code predictor picks codes 1 to 15 one after another. SGLang-Omni runs the code predictor inside the talker’s own step, which is why a frame never makes the talker wait.
code2wav’s chunks. SGLang-Omni hands code2wav 10 frames at a time (codec_coalesce_frames=10 in its Qwen3-Omni config), and each pass also rereads up to 25 earlier frames so the seams between chunks stay smooth. Step 3 leaves the rereading out.
What the talker reads. For each token the engine picks, SGLang-Omni sends the token id to the detokenizer and the token’s embedding to the talker. The talker’s prompt also carries the engine’s view of the photo and the question, which step 3 leaves out. Once the engine’s text runs out, the talker reads a pad token each step and carries on.
The talker’s wait. SGLang-Omni’s talker scheduler (talker_scheduler.py) holds a decode batch until every request in it has its next input. Steps 5 and 6 draw when tokens arrive and when the engine is busy as an illustration, not a measurement: how often the talker waits depends on how busy the engine is.