Project
Open-source C/C++ library for running LLM inference locally on CPUs and GPUs, using the GGUF model format; includes tools like llama-server and llama-cli.
Nemotron 3.5 Super VL, NVIDIA's 120B model, had GGUF, MLX, FP8 and EXL3 conversions on Hugging Face a day after its October 15 release. The roughly 128 GB memory figure comes from a DEV Community write-up whose author did not run the model.
Reality
NVIDIA says its Nemotron 3.5 Bolt 74B-A9B agent model fits in 24GB and will ship with open weights, data and recipes. The footprint, the license terms and the quality claims stay unverified until the weights are live and outside testers have run them.
Perspective Coverage
12 publishersReality
nFOX Lab plans to release Blockhouse, a kernel-level sandbox that removes a self-hosted AI model's privileges by default, under the Apache 2.0 license. For now it is available as a beta for qualified enterprise evaluations.
Reality
Tom's Hardware tested Nvidia's RTX Spark in a 128GB Surface Laptop Ultra against an M5 Max MacBook Pro on agent task time. Its headline calls local AI a strength and gaming a weak spot, but the text reviewed here ends before the AI timings.
Reality
Janas, a C engine by Maurizio Cammalleri, reports 37 tok/s on a 48.4 GB model with 32 GB of RAM, against 2.8 on llama.cpp. Speculative drafting supplies 13 of those 37, so the like-for-like figure is 24 tok/s.
Reality
A 26M-parameter edit tagger scored F0.5 0.799 at phone-keyboard grammar correction against 0.663 for fine-tuned small LLMs, its authors report. It altered correct text 2.4% of the time against 20.1%, on a benchmark the authors ran themselves.
Reality
Ollama's OpenAI-compatible endpoint reported 3,875 prompt tokens per second where the real rate was 125, a dev.to author found. Any benchmark that repeats a prompt and reads that field overstates prefill speed by the share served from cache.
Reality
Tencent has open-sourced its Hy-MT2 translation models for 33 languages, with the smallest compressed to 440 MB to run on devices. The claim that the small model beats Microsoft's API is Tencent's own, so teams paying per call have a candidate to test.
Reality
TornadoVM's University of Manchester team says its jitLLM engine runs LLM inference on NVIDIA GPUs from Java at about 90% of llama.cpp's speed. The figure comes from the project's own RTX 5090 recordings, so a JVM team still needs a run on its own GPUs before it retires a Python sidecar.
Reality
llama-server's default prompt cache restored KV built under the wrong LoRA scale on 10 of 10 test prompts in release b11514. According to the dev.to write-up that found it, servers that swap adapters per request behind a shared system prompt trigger it all day.
Reality
MLCommons' MLPerf Inference v6.1 adds tests that time a whole RAG pipeline and a 1,007-turn coding-agent replay. Those scores help a buyer size hardware only where the buyer's own pipeline spends its time the way the four-model reference does.
Reality
VIDRAFT released a 111GB, 4-bit build of its 180B-parameter Darwin model that it says runs on a laptop with 8GB of VRAM and 32GB of RAM. Air-gapped teams still need a measured token rate on that laptop before they can price it against a server.
Reality
Ollama and llama.cpp added local endpoints within a week for Jev decision models, 144M-to-27B checkpoints that return a label and a probability. Routing calls can move off hosted models once teams check each checkpoint's licence and test it on their own labels.
Reality
Cloudflare released two Jev-API-compatible decision models, Clef and Clef-flash, on Workers AI and as Apache 2.0 weights on Hugging Face. Typed classification steps in agent code can now move between providers or onto owned hardware, as long as they stay inside the text-only, 32k-context features Jev supports.
Perspective Coverage
4 publishersReality
llama-server b11430 reports logprob 0.0 for every speculatively decoded token, dragging one test's mean logprob from -0.48 to -0.0011. Nothing in the response or the server log flags the fill-ins, so evals and calibration built on those numbers go wrong quietly.
Reality
Strata runs a 125B-class Qwen model at about 100 tok/s on one RTX 4090 because its sparse MoE activates only about 6B parameters per token. The design moves the hardware bill to system RAM, and the 100 tok/s figure holds mainly for code and structured output.
Reality
Salvatore Sanfilippo's open-source ds4 engine runs a short list of large open-weight models locally by compressing their routed experts to about two bits. Even compressed, the supported builds need high-memory Macs or GPU systems that most people do not already own.
Reality
llama-server left the request hanging or crashed outright in all 73 trials where a 17,653-token prompt arrived 3 to 46 ms before its sleep timer fired. A one-token warm-up request sent first avoided both failures in 123 of 123 tries.
Reality
VIDRAFT says POCKET-Darwin-180B, a 111 GB 4-bit GGUF build of its 180B mixture-of-experts model, runs in llama.cpp on about $1,400 of consumer hardware. The accuracy evidence so far is one MMLU-Pro comparison that VIDRAFT reports itself.
Reality
llama.cpp merged a /v1/systemone endpoint on October 2 that returns typed answers with probabilities from five open model families running locally. Whether it can replace hosted classification calls depends on calibration that each team has to measure on its own data.
Perspective Coverage
3 publishersReality
Build · October 1, 2026 · Confirmed2 publishers
Build · September 17, 2026 · Confirmed2 publishers