One request, end to end
From an HTTP POST to the GPU and back.
Launch the server. It loads the 16 GB of weights onto the GPU, carves most of the rest of the card into a KV pool, and opens an HTTP port. Then it waits.
The moving parts, once
sglang serve (or the older python -m sglang.launch_server) starts three kinds of process. The HTTP server (FastAPI on uvicorn) and the TokenizerManager share one process: it accepts connections, applies the chat template, tokenizes, and owns every open response stream. The Scheduler runs in its own process, one per GPU, and does nothing but the loop from chapters 6 to 8. The DetokenizerManager turns token ids back into text incrementally.
They talk over ZeroMQ PUSH/PULL sockets on ipc:// Unix domain sockets: TokenizedGenerateReqInput in, BatchTokenIDOutput to the detokenizer, BatchStrOutput back to the server. The scheduler calls recv_pyobj(zmq.NOBLOCK) in a loop at the top of each step to drain everything that arrived. Streaming responses are server-sent events, one data: line per token in the OpenAI-compatible format.
Open connections live in the server process and cost it a little memory each, not GPU time, so a burst of clients does not slow the loop. The scheduler runs as many requests as the KV pool has room for (chapter 7), capped further by --max-running-requests. The rest wait in Scheduler.waiting_queue with their connections open, and see a longer wait for the first token rather than an error. Past --max-queued-requests, the server answers new requests with 503 instead of queueing them.