Topic
The working state written during prefill, appended to during decode, and sharing device memory with the weights.
SparseEngine's paper reports about 10x vLLM decode throughput at each system's largest tested batch size, but 1.5x to 2.6x at matched batch sizes. Plan on the matched range. The paper measured it with Quest and OmniKV.
Reality
llama-server's default prompt cache restored KV built under the wrong LoRA scale on 10 of 10 test prompts in release b11514. According to the dev.to write-up that found it, servers that swap adapters per request behind a shared system prompt trigger it all day.
Reality
Llama 8B on an RTX 4060 Ti 16GB fell from 42.5 to 3.8 tokens a second with a fifth of the model in system RAM, according to a dev.to benchmark. For a local coding agent, that makes VRAM for weights plus context the first spec to check, ahead of bandwidth.
Reality
Qwen Flash-Next NVFP4 ran in vLLM at 131,072 tokens of context and 16 sequences only after loader patches and cuts to its original targets. Its one load test covers only the bfloat16 KV-cache baseline, so operators on the later B12x stack have to run their own.
Reality
Quantized to about 4 GB, a 7B model still runs out of GPU memory near 30,000 tokens as its KV cache grows, according to a dev.to walkthrough. Sizing a card by its weight file alone leaves that growing cache off the memory budget.
Reality
V4.1-Flash retires the V4 Pro line and carries two active-parameter counts, 763B total with 8B on input tokens and 16B on output, so one sizing number no longer covers both phases of a request. Baseten had it running on day zero.
Perspective Coverage
3 publishersReality
OpenAI's guide prices reused input tokens at a discount of up to 90 percent. It also says cached key-value states sit on individual machines, so an unchanged prefix can still miss the cache when routing overflows.
Reality
The weights formula is the easy part of sizing local inference. Quantization metadata, the KV cache and the runtime's own buffers decide whether a 70B model fits, and a dev.to walkthrough shows where the advertised bit width stops helping.
Reality
DigitalOcean's serving guide puts the cost of a long request on two mechanics, the prefill pass and the KV cache, and says the real choice is which mix of caching, prefix reuse and retrieval fits the budget.
Reality
One 8,192-token session on a 27B model holds 512 MB of key-value cache, or 64 KB for every token generated. How many of those sessions fit in free VRAM sets serving concurrency, and paging decides the waste.
Reality
A dev.to guide to running local models on 8GB prices the KV cache between 15KB and 160KB per token depending on architecture. At 32K tokens held, that spread is the difference between 0.5GB and 5GB of a fixed budget.
Reality
Decode is memory-bandwidth bound. Eight concurrent 128k Llama 3 70B streams need 97.8 ms of HBM transfer for every token generated, and DeepSeek's 512-scalar latent is aimed squarely at those bytes.
Reality
nano-vLLM's decode cost fits in one expression, W/B plus KV(Tavg), and a batch-1 context sweep on Qwen3-0.6B shows why those two terms move in opposite directions as you add requests or lengthen prompts.
Reality
Prefill and decode sit on the same card and answer to different limits, which is why a GPU with more arithmetic and the same memory read rate leaves your time per output token exactly where it was.
Reality