Streaming audio decode
Output with a real-time deadline.
The player plays each chunk the moment the one before it ends. If the next has not arrived, the sound stops. Two things matter: how soon the first sound comes, and whether a gap ever opens.
Where this chapter’s claims come from
Gaps. SGLang-Omni’s benchmark (benchmarks/metrics/playback_continuity.py) replays each streamed request as a player would: each chunk plays from when it arrives or when the last one ends, whichever is later, and the request is scored by its longest underrun, the time the player had nothing to play. It reports how many requests keep that under 50, 100 and 200 ms. Step 1’s arrival times are illustrative.
Without streaming. Step 2’s decoder is the simplest one, drawn for contrast. SGLang-Omni itself decodes window by window either way; when a request asks for its sound whole, code2wav keeps each window’s sound and sends it all at the end. The 250 frames are chapter 16’s 20 s answer at 80 ms a frame. Steps 2 and 3 draw the talker at a frame every 40 ms, twice real time, and each code2wav call as 40 ms; both are illustrative.
Inside code2wav’s stage. code2wav’s scheduler is Code2WavScheduler, built on StreamingVocoderBase, the streaming kind of Ch. 17 Steps 7 and 11. For each request it keeps a state (Code2WavStreamState): the frames received so far (chunks) and how many have become sound (emitted). Each time frames arrive it checks whether 10 new ones are in (should_decode); if so it decodes a window and puts the new sound on its outbox as a stream message, which the coordinator merges into the response (Ch. 18 Step 10) and the server sends as events (Ch. 9 Step 9). When the talker’s stream ends, on_stream_done decodes whatever is left even if it is fewer than 10 frames, sends that last piece and the final result, and clears the state; aborted requests have their state dropped, and late frames for them are ignored. Step 4’s request has 34 frames, and its arrival times are illustrative.
The pace. A frame is 80 ms of sound (1,920 samples at 24,000 a second) and 16 codes, one from the talker and 15 from the code predictor, each a step that waits for the one before (Ch. 16 Step 3), so keeping pace means a frame within 80 ms, about 5 ms a step for the whole batch. The 38% is the mean real-time factor in SGLang-Omni’s speech-synthesis test, 16 requests at once on one H100, the worst of five calibration runs; it is measured on whole requests, so it shows the average pace, not that no chunk was ever late.
Joining chunks. code2wav is eight transformer layers that each look back over up to 72 frames, followed by convolutions that turn each frame into 1,920 samples, so each moment of its output depends on earlier frames. Each call runs the whole model on its window and keeps nothing for the next, unlike the thinker, whose KV cache carries a request’s past from step to step. So each window decodes up to 25 earlier frames (left_context_size=25) with the new ones and keeps only the last 10 × 1,920 samples (code2wav_scheduler.py, decode_delta). Step 5’s waveforms are drawn for illustration.
Chunk size. With chunks of N frames, the first sound waits for N frames, and once 25 earlier frames exist each window decodes 25 + N frames to send N. SGLang-Omni’s default is 10 (stream_chunk_size=10); PR #2800, still open, proposes a 4-frame first window (initial_codec_chunk_frames=4) followed by windows of 10. On one H200 running every stage, with 16 SeedTTS requests at once, that change alone moved first-sound p95 from 0.61 s to 0.48 s and requests per second from 12.79 to 12.64.
Not covered. code2wav records a CUDA graph for each window size it will see (10, 20, 30 and 35 frames), and the talker sends its first 10 frames one at a time and later ones in groups of 10 so that every window lands on one of them; PR #1101 measured the graphs at 9–12% more requests per second on an H200. code2wav can also batch windows from several requests into one call, and it copies each chunk’s sound to the CPU while the next window runs.