Project
NVIDIA's open-source library for optimizing and serving large language model inference on NVIDIA GPUs, used in high-performance LLM deployment stacks.
Spotify launched LLM podcast recommendations in January 2026 on lower-tier GPUs because vLLM's beam search left high-end hardware underused. Its post on a collaboration with NVIDIA on TensorRT LLM starts from that constraint.
Reality
NVIDIA and Princeton's Dao AI Lab cache SSM inputs instead of state, targeting a decode step where SSM layers are the largest component up to 100K tokens. ReplaySSM is already integrated into TensorRT-LLM, vLLM and FlashInfer, NVIDIA says.
Reality
NVIDIA measured TensorRT LLM holding 96.1 to 98.2 percent of its non-confidential output throughput on Blackwell, and it got there by unpinning host memory on the affected paths, moving decode readback off the scheduler thread, and timing kernel tactics with the GPU's global timer instead of CUDA events.
Reality
MLPerf Inference v6.1 preview submissions put Vera Rubin NVL72 at up to 3.7x GB300 on Qwen3-VL and up to 2.5x on DeepSeek-R1, on two different inference frameworks. The four-rack 99% scaling result is an offline number.
Reality
SemiAnalysis says its AgentX benchmark clocked Nvidia's Vera Rubin NVL72 at up to seven times Blackwell's token throughput per megawatt on pre-release software, and puts the profit gain at over two times per gigawatt.
Reality
SageMaker Inference now sends requests that begin with the same tokens to the same instance. On AWS's seven-node Llama 3.1 70B benchmark that lifted the KV cache hit rate from roughly 25 percent to 82 percent.
Reality
Qwen3.8-Flash-Next puts 36 Gated DeltaNet layers and 12 sparse-attention layers on Hugging Face, which means the retrieval budget Qwen4 will inherit is something you can measure against your own traces now.
Perspective Coverage
5 publishersReality
Ten Chinese model variants now answer an OpenAI tool-calling payload well enough to return a valid response, which is the easy part. The streaming deltas and call arity behind it are where an agent migration spends its budget.
Reality
Speculative decoding speedups depend on the data, and most published ones come from high-level scripts on narrow datasets. SPEED-Bench's authors argue the honest measurement happens inside vLLM or TensorRT-LLM, across concurrencies.
Reality
The bandwidth gap is 4.4x and the capacity gap is 4x, which is why these two boxes are not really competing. One decides whether a model fits; the other decides whether it is usable.
Reality
NVIDIA Dynamo keeps a pre-warmed engine on the same GPUs and hands it the resident weights instead of reloading them. The measured recovery window falls to about 2.6% of a cold restart.
Reality
SemiAnalysis's AgentX replays recorded coding-agent sessions instead of fixed prompts. On that traffic, the reported Nvidia-AMD cost gap grows as the interactivity target rises.
Reality
SemiAnalysis has pushed fixed-length serving into maintenance mode. On replayed Claude Code traffic, NVIDIA's GB300 is credited with 15x Hopper, against 40x over H200 on the retired static test.
Reality
The update adds a path selector and a two-tap convolution rather than layers, recovering most of the accuracy that tripling the drafter bought at 15.2% latency, by the vendor's own numbers.
Reality
A Kubernetes serving guide makes a point most teams skip: the YAML you reviewed is identical across five engines, and everything it hides is what breaks in production.
Reality