Topic
Field covering engines (vLLM, SGLang, Ollama) and techniques like KV caching, PagedAttention, and batching that run trained LLMs efficiently in production.
Spotify launched LLM podcast recommendations in January 2026 on lower-tier GPUs because vLLM's beam search left high-end hardware underused. Its post on a collaboration with NVIDIA on TensorRT LLM starts from that constraint.
Reality
BAAI's AREX-2 config declares one multi-token-prediction layer, but its weight index lists 0 such tensors against Qwen3.8-27B's 15. Its dev.to testers say the base-model drafter and quant need rechecking too, though they have not run AREX-2.
Reality
EcoHash reports that one vLLM flag took Qwen3.8-27B on a single RTX Pro 6000 from 999.9 to 1954.0 output tokens/sec. The figure comes from its own production config on vLLM 0.27.1, so it is one team's measurement on its own workloads.
Reality
JFrog disclosed an unauthenticated remote code execution flaw in LMCache's multiprocess mode, CVE-2026-105192, rated CVSS 9.8 with no patch. A public proof of concept exists, so any deployment that exposes the cache server to a network needs access limits or the mode turned off.
Reality
vLLM 0.32.0rc0 sets the --allowed-origins default to an empty list, where 0.31 and every release back to 0.6 allowed all origins. A dev.to write-up says browser UIs on another origin will break on upgrade while server-side clients keep working.
Reality
Serving a 70B model takes about 141 GB of GPU memory for weights at 16-bit, 73 GB at FP8 and 40 GB at 4-bit, according to a dev.to post. Fine-tuning takes about 50 GB with QLoRA and 1.1 TB with full AdamW. The post maps both onto GPU setups and hourly rental prices.
Reality
Qwen released eight open-weight Qwen3 models, two MoE and six dense, that each support thinking and non-thinking modes. On the model card, one chat-template argument, enable_thinking, picks the mode and defaults to True.
Reality
NVIDIA and Princeton's Dao AI Lab cache SSM inputs instead of state, targeting a decode step where SSM layers are the largest component up to 100K tokens. ReplaySSM is already integrated into TensorRT-LLM, vLLM and FlashInfer, NVIDIA says.
Reality
llama-server's default prompt cache restored KV built under the wrong LoRA scale on 10 of 10 test prompts in release b11514. According to the dev.to write-up that found it, servers that swap adapters per request behind a shared system prompt trigger it all day.
Reality
A dev.to walk-through pins the change to line 21 of setup.py: no try, no feature flag. Build vLLM from source and you own a Rust toolchain, plus a protoc you were never told about.
Reality
Design notes for a QoS proxy make a narrow, useful argument: on a fixed GPU pool, tier promises are only real if something admits or sheds requests before the engine sees them.
Reality
vLLM 0.30.0 ignores the per-module rank and alpha patterns in PEFT LoRA adapters, and one test put the error against PEFT at 70 times the unpatterned baseline. The adapter loads without complaint, so only a config check or a side-by-side against PEFT will catch it.
Reality
vLLM 0.30.0's weight cache fingerprints only safetensors headers, letting an engine on an altered Qwen3-0.6B run the daemon's original weights. Fine-tunes share their base model's headers, so a mix-up there would produce fluent wrong answers under a log line reporting success.
Reality
Helion's autotuned GEMM beat vLLM's default CUTLASS and DeepGEMM backends on Hopper GPUs, by more than 10% throughput on some workloads, its authors report. The gain rests on per-shape tuning that can run for hours, a cost teams pay in place of kernel maintenance.
Reality
KAIST and Seoul National University's AgSpec nearly doubles accepted draft length for coding agents by indexing files in the diff and JSON forms agents emit. It lives entirely in the retrieval index and leaves model weights untouched.
Reality
Microsoft released metadata from 301,026 GitHub Copilot agent sessions, covering 9.3 million LLM calls in one June week. Capacity planners now have real cache and token figures to test against, though the files measure resource use only and cannot show whether the output was any good.
Reality
SageMaker costs 1.40x a plain EC2 instance per hour to serve one Gemma 4 vLLM build on the same T4 or L4 GPU, a benchmark on dev.to finds. With decode speed matched within 2%, the extra 40% goes to the managed layer around the GPU.
Reality
Gemma 4 decodes on SageMaker's smallest GPU, a T4, at 0.8x an L4's speed with matching outputs from a patched vLLM image, a dev.to benchmark reports. The T4 is the cheaper choice per token only when it rents for under 80% of the L4's hourly rate.
Reality
Repacking Gemma 4's QAT weights with 4-bit embedding tables made decode up to 1.39x faster on one SageMaker L4, according to a dev.to benchmark series. For teams serving Gemma 4 on vLLM, how the weights are stored becomes a setting to measure alongside model size.
Reality
Qwen Flash-Next NVFP4 ran in vLLM at 131,072 tokens of context and 16 sequences only after loader patches and cuts to its original targets. Its one load test covers only the bfloat16 KV-cache baseline, so operators on the later B12x stack have to run their own.
Reality
Build · September 15, 2026 · One report1 publisher
Build · August 31, 2026 · One report1 publisher