Sharing GPUs
Several models, one card.
Qwen3-Omni's default layout takes two GPUs. GPU 1 holds only the talker, 6.6 GB of weights on an 80 GB card. Why not move the talker onto GPU 0 and serve from one?
Where this chapter’s claims come from
The layouts. SGLang-Omni’s Qwen3-Omni config (models/qwen3_omni/config.py) builds both speech layouts from one function, _speech_stages(thinker_gpu, talker_gpu, …): the default speech variant passes talker_gpu=1, speech-colocated passes talker_gpu=0. Every other stage keeps its device. Its placement policy (models/qwen3_omni/placement.py) refuses a thinker and talker on one GPU unless the colocated config is used. The design notes (docs/design/refactor_rfc.md) say earlier versions rejected same-GPU placement of the speech stages outright.
Why two GPUs by default. Steps 4 and 5 give two reasons. The second is SGLang-Omni’s own, measured; the first is worked out from its numbers. Step 4 is the least memory its video test could run with on one H100. The thinker’s weights are 61.0 GB. The thinker sets 5% of the card aside for the video encoder’s peak working memory (encoder_mem_reserve, default 0.05). SGLang will not start a request its KV cache cannot hold whole (Ch. 7 Step 1), and the thinker accepts requests up to 32,768 tokens, 3.1 GB at 96 KB a token; a Video-MME prompt averages 13,769 tokens in SGLang-Omni’s benchmark over 2,496 videos, and the longest approach that limit. The encoders, code2wav and talker are given their shares from step 3’s one-H100 config, the smallest SGLang-Omni ships, and 6% of the card is held back for CUDA graphs, workspaces and peaks, as in that config. The total is 85.7 GB. Step 3’s config itself gives the thinker only 78%, 62.4 GB, which is why it is written for speech-synthesis tests, whose prompts are a few hundred tokens. SGLang-Omni’s record agrees: when its tests moved from an H20 (96 GB) to H100s (PR #883), its notes say the one-card bf16 setup did not fit, and its video test with spoken answers (16 videos at once, up to 128 frames each, bf16) runs on two H100s in the default layout. Until PR #430 (May 2026) it rejected the thinker and talker on one GPU at all, so the two-GPU layout also came first. Its one-GPU configs are for the H20, the H200 (141 GB), FP8 weights on an H100, and its bf16 speech-synthesis tests on an H100.
Taking turns. PR #1235 (August 2026) measured the talker’s placement on H200s with 8 requests at once and the thinker split over two GPUs, changing only whether the talker shared a GPU with one of the thinker’s halves or had its own. On its own GPU the talker kept it 86% busy (median, nvidia-smi utilization: the share of time some kernel is running). Sharing made the slowest tenth of first sounds about twice as late (p90 2.70 and 2.66 s against 1.46 and 1.42 s, two runs each), the whole run 9–16% longer and the time spent stalled about twice as long; the PR notes that single-run changes of 10–30% were within that host’s noise, which the p90 and stall numbers are well outside. With one request at a time, the talker on its own GPU got first sound about 30% sooner but finished 14–17% later, since the thinker’s tokens then cross between GPUs. The same PR found code2wav only 9–13% busy and moved it onto the thinker’s GPU, which is the layout step 1 shows. The PR’s shared case is the talker beside half of the thinker, not all five stages on one GPU. NVIDIA’s MPS, which SGLang-Omni turns on with --mps auto (off by default), lets kernels from several processes run at the same time instead of taking turns; SGLang-Omni publishes no Qwen3-Omni measurement with it.
Gigabytes. Like chapter 7, this chapter counts an H100 as 80 GB. The card holds 80 GiB, which PyTorch reports as about 85 GB, and a share is a fraction of what it reports. Counted that way, step 4’s total is 87.1 GB against 85, still 2.1 GB too many.
The weights. Parameter counts are summed from the tensor shapes in the Qwen/Qwen3-Omni-30B-A3B-Instruct checkpoint: 30.5B for the thinker with its output layer, 3.3B for the talker with its code predictor, 0.65B and 0.54B for the audio and vision encoders, 0.22B for code2wav. At 2 bytes a parameter that is 70.5 GB, the checkpoint’s size.
The shares. The numbers in step 3 are gpu_memory_fraction in examples/configs/qwen3_omni_colocated_h100_bf16.yaml. A share is a fraction of the card’s whole memory, not of what is free, so it does not depend on which stage loads first. For the thinker and talker, SGLang’s KV cache gets the share minus what the stage itself has already allocated (design notes, PR #430); step 4 works out what the thinker needs for video. The planner (config/placement.py) adds up the shares on each GPU and refuses a GPU past its limit, 100% by default. A share is checked at placement and not enforced while serving, unless the stage declares a byte budget, in which case its process’s PyTorch allocator is capped so that a stage that outgrows its budget fails itself rather than a neighbour.
Not covered. Stages on one GPU are also built one at a time, behind a lock per GPU, so each measures memory after the ones before it have loaded. SGLang-Omni can also run several copies of a stage, or of the whole server, on one GPU (replica_devices), optionally sharing one copy of the weights. Its notes on CUDA MPS report 1.4–2.1× the throughput of one copy for two or three copies of a speech-synthesis model on one GPU; they publish no such measurement for Qwen3-Omni, so step 5 is drawn as an illustration.