Chapter 15

Speculative decoding

Guess several tokens, verify in one step.

inside one pass · the weights arrive a layer at a time · the first 6 of 32 layers1 position · normal decoding, the newest tokenmemorycomputelayer 1layer 2layer 3layer 4layer 5layer 6×Wx · 1 × 4,096, a row per positionW · 4,096 × 14,336 = 117 MB5 positions · 5 tokens in one pass, as prefill doesmemorycomputelayer 1layer 2layer 3layer 4layer 5layer 6×Wx · 5 × 4,096, a row per positionW · 4,096 × 14,336 = 117 MBlayer 1's math runs while layer 2 is read, then waits5 rows make each burst of math 5 times longer, and it still ends before the next layer arrivesso the pass ends when the read ends: 4.8 ms, for 1 position or 5W: one of a layer's weight matrices (the MLP's gate) · the pass: 4.8 ms of reading, 0.016 ms of math a position (Ch. 4 Step 5)the math is drawn about 22× wider than scale so it shows

A position is one token the pass works on, one row of numbers. Each layer's weights arrive once and every row is multiplied by them while the next layer is read, so 5 rows make the math longer but not the pass.

← Attention backends
01 / 08
Where this chapter’s claims come from

The acceptance rule. With greedy decoding a draft token is kept when it equals the token the target would have picked. When sampling, each draft token is kept with probability min(1, p_target / p_draft), and the first rejected position is resampled from a corrected distribution, so the output follows the target’s distribution exactly (Leviathan et al., 2023; Chen et al., 2023). Steps 1 to 3 time one request alone and leave out the KV cache read, which is small at this context length (Ch. 4 Step 6). A pass takes as long as the longer of its read and its math (Ch. 4 Step 5), because each layer’s math runs while the next layer’s weights are being read; five positions’ math is 0.08 ms in all, so a pass over five positions still takes the 4.8 ms read.

EAGLE. EAGLE drafts with one Transformer layer that reads the target’s own hidden states; EAGLE-3 fuses hidden states from a low, a middle and a high layer and trains the head on its own drafts (“training-time test”). SGLang runs it with --speculative-algorithm EAGLE3; for Llama models its defaults are 5 draft steps, 4 branches per step and 8 draft tokens verified (SGLang’s speculative decoding docs).

DFlash. Chen, Liang and Liu, DFlash: Block Diffusion for Flash Speculative Decoding (ICML 2026, arXiv 2602.06036). The drafter has 5 layers and drafts a block of 16 positions in one pass; hidden states from 5 target layers are projected into one context feature and added to the keys and values of every draft layer. SGLang runs it with --speculative-algorithm DFLASH.

DSpark. Cheng, Yu, Shao, Li, Xiong et al. (DeepSeek-AI and Peking University), DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation (arXiv 2607.05147). Its parallel stage is a DFlash backbone; the sequential stage shown here is its Markov head, a low-rank (rank 256) bias from the previous draft token. The confidence head and the hardware-aware prefix scheduler are its Sections 3.2.1 and 3.2.2. DeepSeek reports 60–85% faster per-user generation on DeepSeek-V4-Flash than its previous one-token MTP drafter.

The comparison. Normal decoding’s 1 token per target pass is the baseline. Step 8’s other numbers are DSpark’s Table 1 for a Qwen3-8B target: all three drafters retrained on the same data, a draft length of 7, chain drafting, temperature 1, and the accepted length counting the target’s own token. The DFlash paper’s own comparison against EAGLE-3 (tree drafting, temperature 0) reports larger gaps; setups differ, so the numbers are not interchangeable.