Chapter 10

Scheduler deep dive

One Req through one step.

Inside process 2. Ch. 6 Step 9's choice of batch is one of this loop's six stages. Around them: two lists of Req objects, the page pool and radix tree (Ch. 7 Step 4, Ch. 8 Step 4), and a model runner that owns the GPU.

← One request, end to end

01 / 11

The loop, once, as code

while True:
    ingest_requests()                      # drain the zmq inbox, non-blocking: message → Req, waiting_queue.append
    batch = get_next_batch_to_run()        # merge last step's prefill into running, match prefixes, take pages:
                                           # a prefill batch if anyone waits and pages allow, else running_batch
    if batch is None:
        on_idle()
        continue
    result = run_batch(batch)              # ScheduleBatch → ForwardBatch → ModelRunner.forward → Sampler
    process_batch_result(batch, result)    # append tokens, finish Reqs, free/keep pages, send BatchTokenIDOutput

That is Scheduler.event_loop_normal in managers/scheduler.py. event_loop_overlap is the same loop with the CPU work of the next step launched before the current step’s result is processed; sampled tokens are left on the GPU and read by the next forward pass directly, which is why the GPU lane stays solid.