Chapter 18

Between stages

Eight edges, four ways to cross them.

what crosses each edge of Ch. 17 Step 12's graph · how big, and how often12345678preprocessingCPUimage_encoderGPU 0audio_encoderGPU 0thinkerGPU 0decodeCPUtalker_arGPU 1code2wavGPU 0#edgewhat crossessizehow often1preprocessing → image_encoderthe photo, as patches896 × 896 × 3 = 2,408,448 valuesonce2preprocessing → audio_encoderthe question, as frames10 s × 100 = 1,000 feature framesonce3preprocessing → thinkerthe prompt's token idsone number per text positiononce4image_encoder → thinker784 embeddings784 × 4 KB = 3,136 KBonce5audio_encoder → thinker130 embeddings130 × 4 KB = 520 KBonce6thinker → decodea token idone numberevery token7thinker → talker_arthe token's vector2,048 × 2 B = 4 KBevery token8talker_ar → code2wavcodes10 frames × 16 = 160 numbersevery 10 frames9–11the same inputs → talker_arrows 3–5, for its promptas rows 3–5once4 KB: one vector is the thinker's width, 2,048 numbers of 2 bytes · the photo and the question: Ch. 16 Steps 1 and 2edges 9–11 are not drawn: talker_ar builds its own prompt from the same inputs as the thinker

Ch. 17 Step 12 numbered the edges. What crosses them ranges from one token id to 3,136 KB of embeddings, sent once per request or once per token.

← Stages and schedulers
01 / 11
Where this chapter’s claims come from

The transports. SGLang-Omni’s CommRouter (comm/router.py) picks a transport for each edge: a plain reference when both stages share a process, Mooncake when the receiver is on another machine, CUDA IPC when the data is on a GPU and both stages run on GPUs in one machine, and shared memory otherwise. CUDA IPC between two GPUs needs the GPUs to reach each other’s memory; if they cannot, the router falls back to shared memory. The edge-by-edge transports in step 9 follow these rules for the default speech layout; they were not traced on a running server.

The hand-off. The sender’s runtime writes the data through a relay, then sends a DataReadyMessage over ZMQ naming the request, the two stages and where the data is. The receiver reads it, sends a DataAckMessage so the sender can reuse the space, and puts it in its scheduler’s inbox. The CUDA IPC relay keeps a fixed pool of slots on the sender’s GPU; the ack frees a slot.

The sizes. One vector is the thinker’s width, 2,048 numbers, at 2 bytes each in bf16. Each encoder embedding has the same width, since it takes a position in the thinker’s prompt. Qwen3-Omni also passes a few extra vision layers to the thinker, which the table leaves out.

The 33%. SGLang-Omni’s design notes (PR #496) report that its text-only pipeline, which runs every stage in one process, cut its mean latency by 33% when its stages began passing references instead of going through the relay.

What is not Omni’s own. CUDA IPC, shared memory and RDMA transfer engines are general tools. SGLang also uses Mooncake to move KV cache between servers in prefill-decode disaggregation (chapter 28). What SGLang-Omni adds is the choice per edge, from where the edge’s two stages sit.