sgl-project /

SGLang

Live map
View repository
Snapshotmain · d3589a7

How serving requests become continuously scheduled batches that reuse prefix KV state and stream decoded tokens.

Click a building or route
Choose a path

Tokenize a new prompt, allocate its KV state, run prefill, and stream the first token.

System map8 components · 9 connections
130%
SGLang architectureHTTP and tokenization begin at left. Scheduling, batching, and the radix cache fill the center; tensor-parallel execution and detokenization complete the flow on the right.prompt, media, samplin…tokenized generate req…prefill and decode req…input token sequencecache indices and reus…distributed model work…tokens, positions, cac…output token IDstext delta and finish …HTTP serverTokenizermanagerModel runnerTensor-parallelworkerSchedulerDetokenizermanagerSchedule batchRadixprefix cache
Connections
BuildingsEntryServiceComputeStorageRuntimeToolingExternal

Terms, in plain English

Radix cacheA prefix tree mapping token sequences to reusable KV cache entries.

How to study this repo

  1. 1
    Trace one request

    Follow HTTP input through tokenization, scheduling, TPWorker execution, detokenization, and streaming.

  2. 2
    Compare prefix hit and miss

    Read scheduler and radix-cache state changes for two prompts that share a long prefix.