vllm-project /

vLLM

Live map
View repository
Snapshotmain · fe1c317

How an API request enters continuous batching, receives paged KV memory, runs on a GPU, and streams tokens.

Click a building or route
Choose a path

Admit a prompt, allocate its cache blocks, run prefill, and stream the first sample.

System map7 components · 8 connections
130%
vLLM architectureThe API and asynchronous engine sit left, scheduling and KV state occupy the center, and the GPU runner plus output path finish on the right.prompt, sampling param…EngineCoreRequestrunning and waiting re…token counts and cache…scheduled tokens and b…paged attention statetoken IDs and log prob…text, usage, finish re…OpenAIAPI serverAsyncLLMGPU modelrunnerEngineCoreOutputprocessorContinuousschedulerPaged KV cache
Connections
BuildingsEntryServiceComputeStorageRuntimeToolingExternal

Terms, in plain English

PagedAttentionAttention over KV state stored in non-contiguous fixed-size blocks.

How to study this repo

  1. 1
    Follow one completion

    Trace an API request through AsyncLLM, EngineCore, Scheduler, GPUModelRunner, and OutputProcessor.

  2. 2
    Account for KV blocks

    Compare a prefix-cache hit and miss to see how scheduled token counts and block allocation change.