Chapter 09

One request, end to end

From an HTTP POST to the GPU and back.

python -m sglang.launch_server --model-path … --port 30000http serverlistening on :30000async · thousands of connectionstokenizertext ↔ idsthe scheduler loopGPU · 80 GBweightsKV pool · 64 GB, emptywaiting list: empty · running: nonestepping 209× a second, doing nothing yetweights loaded once; the KV pool is the 64 GB from chapter 7; the loop from chapter 6 is already spinning

Launch the server. It loads the 16 GB of weights onto the GPU, carves the rest of the card into a KV pool, and opens an HTTP port. Then it waits.

← Prefix caching
01 / 10
The moving parts, once

python -m sglang.launch_server starts three kinds of process. The HTTP server (FastAPI on uvicorn) and the TokenizerManager share one process: it accepts connections, applies the chat template, tokenizes, and owns every open response stream. The Scheduler runs in its own process, one per GPU, and does nothing but the loop from chapters 6 to 8. The DetokenizerManager turns token ids back into text incrementally.

They talk over ZeroMQ PUSH/PULL sockets on ipc:// Unix domain sockets: TokenizedGenerateReqInput in, BatchTokenIDOut to the detokenizer, BatchStrOut back to the server. The scheduler calls recv_pyobj(zmq.NOBLOCK) in a loop at the top of each step to drain everything that arrived. Streaming responses are server-sent events, one data: line per token in the OpenAI-compatible format.