Topic
The practice of running AI models on very long input prompts, often needing specialized memory and compute techniques to control cost, latency, and cache size.
The quarter-size global KV cache and eighth-size persistent storage are serving-cost claims rather than benchmark scores, and banking the second one requires a prefix-cache tier your stack has to already run.
Perspective Coverage
6 publishersReality
Moonshot AI's Kimi Linear reports 75% less KV cache and 6.3x faster decoding than MLA at 1M tokens, using three linear layers per attention layer. The speedup falls to parity at 4k tokens, so the saving goes to traffic that runs at hundreds of thousands of tokens.
Reality
Alibaba's Qwen3.8-Flash-Next preview activates 6B of its 125B parameters per token. Per-token compute drops to under a quarter of the dense 27B's, and about 125GB of weights still has to stay on device.
Reality
The model card for this 552B-parameter Mixture-of-Experts release puts the global KV cache at 890 bytes per token, about a quarter of the previous Flash generation, and every figure in it is DeepSeek's own.
Reality
DeepSeek's V4.1 Flash claims a million-token context on an FP4 KV cache and Alibaba opened 2.4 trillion parameters. The seven reports collected around the two launches quote no latency, throughput or hardware.
Reality
Alibaba's largest open-weight release fits on a single eight-GPU node only because a community four-bit build takes it down to 1.2 TB. AWS publishes the vLLM config for it and skips the price.
Reality
The efficiency claims are self-reported and measured against DeepSeek's own prior model, but the weights are MIT-licensed and already pulled 1.78 million times, which is what turns a ratio into a number a buyer can carry into a renewal.
Reality
Alibaba's 27B model fits a 32GB card with 15GB to spare. Tom's Hardware still had to pick between llama.cpp's full 262K window at half-hour prefill and a supported vLLM deployment capped at 32K.
Reality
Qwen3.8-Flash-Next puts 36 Gated DeltaNet layers and 12 sparse-attention layers on Hugging Face, which means the retrieval budget Qwen4 will inherit is something you can measure against your own traces now.
Perspective Coverage
5 publishersReality
The MIT-licensed 320B model card claims it beats GLM-5.2 at a tenth of the price and approaches Claude Opus 4.8 on coding, but it names no dollar rate, and the comparisons are largely the vendor's own.
Reality