Chapter 10

The scheduler's loop

One Req through one step.

process 2 · the scheduler · one step of the loopinbox · zmq PULLwaiting_queueReqs not yet prefilled · policy orderemptyrunning_batchmid-answer · one token per stepReq · decodingReq · decodingoutbox · zmq PUSHrecv_requestsinbox → Req objectsget_next_batch_to_runprefill or decode batchScheduleBatch → tensorsids · positions · pagesModelRunner.forwardembed → 32 layers → logitsSamplerlogits → tokensprocess_batch_resultappend · finish · sendRadixCacheshared prefixes · chapter 8TokenToKVPoolpages · chapter 7GPU · ModelRunnerweights + kernels · chapters 2–4forward pass · 4.8 ms

Inside process 2. Two lists of Req objects, waiting and running. The page pool and the radix tree from chapters 7 and 8. A model runner that owns the GPU. And one loop, top to bottom, about 208 times a second.

← One request, end to end
01 / 10
The loop, once, as code
while True:
    reqs = recv_requests()                 # drain the zmq inbox, non-blocking
    process_input_requests(reqs)           # message → Req, prefix match, waiting_queue.append
    batch = get_next_batch_to_run()        # a prefill batch if anyone waits and pages allow, else running_batch
    if batch is None:
        continue
    result = run_batch(batch)              # ScheduleBatch → tensors → ModelRunner.forward → Sampler
    process_batch_result(batch, result)    # append tokens, finish Reqs, free/keep pages, send BatchTokenIDOut

That is Scheduler.event_loop_normal in managers/scheduler.py. event_loop_overlap is the same loop with the CPU work of the next step launched before the current step’s result is processed; sampled tokens are left on the GPU and read by the next forward pass directly, which is why the GPU lane stays solid.